How to turn Twitter lists into email digests using Simplescraper, AI and Airtable
Twitter is a great tool for tapping into real-time data and expert advice, and Twitter lists are an excellent way of curating topics that matter to you. However, given the volume of tweets, it's not easy to stay on top of the valuable information being shared.
What if, instead of having to check Twitter every few hours - or minutes, for the addicts - you could receive a summary of your favorite Twitter lists in your inbox every day? No need to chase news and updates, instead a briefing of all the topics that you care about is delivered to you on-demand in an easily readable format.
Sounds like a great time (and scroll) saver. So in this guide we'll walk through the steps to transform Twitter lists into email summaries using Simplescraper's new AI enhance feature and Airtable. It's super easy, with no code and no Zapier/Make duct tape required.
As an example of what we'll create, here's how this Twitter list which covers geopolitics looks after it's been summarized and mailed to your inbox:
Lets get started.
Choosing Twitter lists
First we need to choose a Twitter list.
In this guide we'll use Twitter lists covering Artificial intelligence, Crypto, and OSINT (open-source intelligence) as examples. Each of these topics generate a lot of interesting conversation and news moves fast, so they're ideal candidates for summarization.
Here are the lists which have been created by Simplescraper:
AI: https://twitter.com/i/lists/1678688265367429122
Crypto: https://twitter.com/i/lists/1680830886181322759
OSINT: https://twitter.com/i/lists/1500475604353032198
Of course, you can use your own Twitter list, another person's list, or explore some of the many useful lists that others have created here: https://twitter.com/i/lists/search?q=nocode
Scraping your Twitter lists
Now that we've chosen our Twitter lists, the next step is scraping them. Simplescraper makes this easy. Click here to view the Twitter list scrape recipe and save it to your Simplescraper account. Once it's saved, the next step is adding the URLs to the crawler.
First let's update the URLs so that we can extract the data more easily, and without having to log in to Twitter. To do that we update the domain, swapping "twitter.com" for "nitter.net".
When updated the URLs should now look like this:
AI: https://nitter.net/i/lists/1678688265367429122
Crypto: https://nitter.net/i/lists/1680830886181322759
OSINT: https://nitter.net/i/lists/1500475604353032198
Paste these into the crawler on the Simplescraper dashboard and then we're on to the next step.
Summarizing the lists using AI
In this step we'll instruct Simplescraper's AI enhance feature to summarize the Twitter lists in the most useful way possible. In the Simplescraper dashboard, click on the AI enhance tab and toggle the activate switch to on.
Next, set the run mode to auto and then choose a prompt. Simplescraper provides a 'summarize text' prompt which works nicely however let's get a little more specific and choose 'custom prompt' from the dropdown.
This allows you to prompt Simplescraper AI exactly how you wish, and better customize it to match the scraped data that's being analyzed. Here's a prompt we've found works nicely on Twitter lists:
"You are providing an intel brief to an audience. Your goal is to be brief and insightful while retaining key entities and facts. Summarize the main news items and developments from the text.
- Identify and summarize the most important articles
- Ignore short or repetitive info
- Focus on a variance of stories instead of repeating a single story or topic
Understand that you are reading tweets thus slang may be used or context may be missing. Do you best to understand and communicate the context"
Paste this into the custom prompt text area and then scroll down to the preview section.
In the included properties section toggle the text property on and then click on the generate example button. Simplescraper will analyze the Twitter data using AI, apply the prompt and generate a summary for you.
If everything looks good it's time to send the data to Airtable.
Sending the results to Airtable
Click on the Integrate tab and in the Airtable section toggle Activate to on. Enter your Airtable API key, base ID, and the table name that you wish to save the data to. Under included properties you will find a field called 'AI_enhanced' - this contains the AI generated summary so toggle this to on.
Once that's done, create a single-line text field called 'AI_enhanced' In your Airtable table.
To ensure everything's working, click the test button and after a few seconds you should see a test passed message appear.
For a step by step guide to the Airtable integration please see this guide.
And that's all the steps in Simplescraper done! Simplescraper will now scrape the Twitter lists for data, use AI to summarize the important updates, and send the data to Airtable - all automatically, with no code required.
Now let's jump over to Airtable to create an automation that sends the Twitter summaries to your email inbox.
Sending the results to your inbox (via Airtable automations)
The final step is straightforward: each time Simplescraper adds a field to Airtable containing the AI-generated Twitter list summary, send that summary to your email address.
To accomplish this, navigate to the Airtable table that contains the data you've just sent from Simplescraper and do the following:
- Click Automations at the top of the page, then click 'create new automation' in the left sidebar
- Choose 'when a record is created' as a trigger
- In configuration in the right sidebar, select the table containing the AI enhanced data
- Click 'add advanced logic or action' button in the center of the page
- Choose the send email action
- In the To input box in the right-hand menu, enter your email address
- In the Summary input box, enter a subject for the email
- In the Message input box, select the 'AI_enhanced' field
- Click the Generate a preview button
- If you're happy with how it looks, toggle the automation on at the top of the page as the final step
For more in-depth guides on Airtable Automations check out this guide.
Start summarizing
And we're done! Everything's set up so all you have to do now is run your recipe and Simplescraper will extract the data, summarize it and send it to your inbox via Airtable.
In a few simple steps you've transformed a firehose of Twitter data into a readable summary of key information that lands in your inbox automatically. For daily updates, why not set a schedule to receive summarizes at a preferred time, for example first thing in the morning.
And staying updated on a wider range of Twitter lists is as simple as adding more list URLs to the crawler - each will be summarized and emailed to you automatically.
This is just one example of how AI can be used to enhance the data that you scrape. Keep a look out for more guides on using Simplescraper's AI features to save you time and help you make sense of data, or install the extension and see what you can build.
Happy scraping.
Links mentioned
https://simplescraper.io/dashboard - Simplescraper dashboard
https://simplescraper.io/?sharedRecipe=E4fZfRtziBqwuT8ro28v - Twitter scraper recipe
https://simplescraper.io/docs/scraping-data-into-airtable/ - Simplescraper's Airtable integration guide
https://support.airtable.com/docs/send-an-email-action - Airtable email action guide
https://simplescraper.io/docs/scheduling-webscraping/ - setting a schedule on Simplescraper