Scraping data behind a login

Access protected content by using your credentials or browser cookies.

Browser scraping needs no setup for this: the extension scrapes with your own logged-in session, so any page you can open, you can scrape. Cloud runs happen on Simplescraper's servers, where you are not signed in, so they log in with one of two methods: cookies or credentials (username and password).

Cookies are the preferred method. Some sites expire cookies quickly or don't use them for logins, and the credentials method suits those.

Credentials method

The credentials method uses your username and password to login on your behalf and extract the data that you need. You can create a credential for a website and use it to login to all of your scrape recipes that share that domain. For example, if you have five LinkedIn recipes, you can create a single credential for LinkedIn that will work for all your LinkedIn recipes.

Once a credential is added, it is available under 'Credentials' in the Manage Recipes section. Your password is encrypted, securely stored and not visible to Simplescraper.

Add credentials while creating / editing recipe

• Add a new recipe or edit an existing recipe (see this guide for how to) and scroll down to the Credentials section under Advanced Options
• Click the 'add new credential +' button and add your credential details in the popup that appears. Provide a credential name that is clear and obvious, for example 'LinkedIn login'. Then enter the remaining URL, username and password details
• Click save credentials and wait for the 'Credentials added successfully' confirmation text and then close the popup
• The credential that you created will now be visible in the dropdown list - select it and then save your scrape recipe

Add credentials via Credentials menu

• In the sidebar of the Simplescraper dashboard click Manage Recipes and then choose Credentials Manager and then click the green 'add credential +' button
• In the popup that appears add your credential details. Provide a credential name that is clear and obvious, for example 'LinkedIn login'. Then enter the remaining URL, username and password details
• Click save credentials and wait for the 'Credentials added successfully' confirmation text
• Now when you add a new recipe or edit an existing recipe you will see the credential that you created in a dropdown list in the Credentials section

Cookies method

Add cookies with the Simplescraper extension (preferred method)

1. Save the recipe in the extension, then open the site in a tab and sign in.
2. In the same tab, open the extension, go to 'My recipes' and click the recipe.
3. Under 'Settings', find the 'Cookies' row and click 'Add cookies'. If Chrome asks for permission to read cookies, allow it.

The row now shows how many cookies were saved, and cloud runs of the recipe see the site as you do when signed in. The tab must be on the recipe's own site; otherwise the panel offers to open it for you.

When the cookies expire, sign in again and click the refresh icon on the 'Cookies' row to save fresh ones. 'Remove' stops cloud runs using your sign-in on the site.

Adding Cookies via the Copy Cookies extension

Scraping a webpage behind a login is achieved by using your login cookies for that site. Retrieving this cookie requires using a 3rd party extension called Copy Cookies. Please review their permissions and privacy policy here.

Here's how to scrape data that is behind a login using the Copy Cookies extension:

• Install the Copy Cookies extension

• Navigate to the website that you wish to scrape and login to the site if you are not already logged in

• Once logged in, click the Copy Cookies extension icon and click the copy button to copy the cookies to your clipboard

• In the Simplescraper dashboard (https://simplescraper.io/dashboard), navigate to your recipe, click edit beside the recipe name and then click 'show advanced options'

• Under the cookies section paste the cookies from your clipboard into the text area (ctrl + v or right-click > paste)

• Save your recipe and then run it. Simplescraper should login on your behalf and scrape the data that you need

• Note: some websites don't use cookies for user authentication and so this method may not always work