Source: https://eventregistry.org/documentation?tab=feedOfArticles # Stream of new articles Get access to the news articles as soon as they are added to Event Registry. The API endpoint allows the user to obtain the full list of news articles added to Event Registry in the last few minutes. The endpoint can be used without any additional filters, which will return all added articles. Alternatively, you can specify filters to return only new articles in a certain language, from specific sources, mentioning specific keywords, concepts, etc. This is the only API call that allows returning more than 100 articles per call (up to 2.000). If more than 100 articles are returned, then correspondingly higher number of tokens are used per call. For example, if 250 articles are returned, then 3 tokens are used. The endpoint is meant to be called repeatedly every one or more minutes, depending on the amount of filters used. You can use various parameters to let us know, which articles you have already received. We strongly suggest use of `recentActivityArticles*UpdatesAfterUri` parameters. They will ensure that you get all the new articles and no duplicates. See Code example 3 for details. **Endpoint:** GET | POST `https://eventregistry.org/api/v1/minuteStreamArticles` **Headers:** Content-Type: application/json #### `Parameters` #### `apiKey` string (required) Your API key #### `recentActivityArticlesMaxArticleCount` integer The maximum number of articles to return. Maximum value specified can be 2.000. The number of used tokens will correspond to the number of returned articles (for 300 returned articles, 3 tokens will be used, etc.). Default value: `100` #### `recentActivityArticles*UpdatesAfterUri` string The actual names of the parameters should be `recentActivityArticlesNewsUpdatesAfterUri`, `recentActivityArticlesBlogUpdatesAfterUri` and `recentActivityArticlesPrUpdatesAfterUri`, depending on which `dataType` parameters you require. Specifying the parameter(s) will only return articles that were added since the articles with the provided URI were added. If you are repeatedly calling the endpoint, using this parameter is the best way to ensure that all matching content is retrieved. #### `recentActivityArticlesUpdatesAfterMinsAgo` integer The parameter can be used to retrieve not only the last minute of articles, but more minutes. When using some filtering parameters below you use can use this parameter to call `minuteStreamArticles` endpoint only every few minutes and still retrieve all added content matching your filters. The value should be at most 240 (4 hours). Note that use of this parameter is less desirable than use of `recentActivityArticles*UpdatesAfterUri` as it can result in non-accurate results due to time differences between servers. #### `recentActivityArticlesUpdatesAfterTm` string The parameter can be used to retrieve articles that were collected after the specified time. The value should be in format `YYYY-MM-DDTHH:MM:SS` and should represent datetime in UTC timezone. The value should be at most 4 hours in the past. Note that use of this parameter is less desirable than use of `recentActivityArticles*UpdatesAfterUri` as it can result in non-accurate results due to time differences between servers. #### `Additional Parameters` Click to see additional parameters. #### `articleBodyLen` integer Set the size of the article body that'll be returned in the response. Use -1 for full article body. Default value: `-1` #### `dataType` string | string[] What data types should we search? news content (default, `news`), press releases (`pr`) or blogs (`blog`). Available values: `news`, `pr`, `blog` Default value: `news` #### `keyword` string | string[] Find articles that mention the specified keyword or a phrase. You can specify at most 60 keywords in a single search. If you specify multiple `keyword` parameters, then only articles that mention all of them will be returned, unless you specify `keywordOper` parameter and set it to `or`. Use the `keywordSearchMode` parameter to control how the keyword is matched; see the [keyword search modes](/documentation?tab=keyword_search_options) section. Example value: `Barack Obama` #### `conceptUri` string | string[] Find articles that mention the concept with a concept uri. You can specify at most 50 concepts in a single search. If multiple `conceptUri` parameters are provided, then only articles that are about all specified concepts will be returned, unless you specify `conceptOper` parameter and set it to `or`. Check [autosuggest methods](/documentation?tab=suggConcepts) to get concept URI value for a specified concept label. Example value: `http://en.wikipedia.org/wiki/World_Health_Organization` #### `categoryUri` string | string[] Find articles that are assigned to a particular category. You can specify at most 20 categories in a single search. If multiple `categoryUri` parameters are provided, then articles that are about any of the specified categories will be returned, unless you specify `categoryOper` parameter and set it to `and`. Check [autosuggest methods](/documentation?tab=suggCategories) to get value for a specified category name. Example value: `dmoz/Business/Accounting` #### `sourceUri` string | string[] Find articles that have been published by a news source. If you specify multiple `sourceUri` parameters, then articles from any of the specified sources will be returned. Check [autosuggest methods](/documentation?tab=suggSources) to get value for a source name. Example value: `bbc.co.uk` #### `sourceLocationUri` string | string[] Find articles that were published by news sources located at the given geographic location (city or country). If you specify multiple `sourceLocationUri` parameters, then articles from sources from any of the specified sources will be returned. Check [autosuggest methods](/documentation?tab=suggLocations) to get value for a location name. Example value: `http://en.wikipedia.org/wiki/Germany` #### `sourceGroupUri` string | string[] Find articles that were published by news sources that are assigned to some predefined group of news sources. Example value: `general/ERtop10` #### `authorUri` string | string[] Find articles that were written by a particular author. Check [autosuggest methods](/documentation?tab=suggAuthors) to get value for author uri based on the author name (and potentially source url). Example value: `mark_mazzetti@nytimes.com` #### `locationUri` string | string[] Find articles that describe something that occured at a particular location (based on the location mentioned in the dateline). Check [autosuggest methods](/documentation?tab=suggLocations) to get value for a location name. NOTE: If a location will not be specified in the dateline (immediate start of the article), the article will not be returned. Consider also using `conceptUri` to find articles that mention a particular location. Example value: `http://en.wikipedia.org/wiki/United_States` #### `lang` string | string[] Find articles in the specified language(s). If parameter is not set, articles in all languages will be included. You can specify at most 5 languages in a single search. Specify languages using their ISO3 language codes. The full list of languages and their ISO3 codes can be found on [this page](https://github.com/EventRegistry/event-registry-python/wiki/Supported-languages). Example value: `eng` #### `keywordLoc` string What data should be used when searching using the keywords provided by `keywords` parameter. Available values: `body`, `title`, `title,body` Default value: `body` #### `keywordOper` string If more keywords are provided with the `keyword` parameter, what should be the Boolean operator used. If `and` (default) then all of the specified keywords have to be present in the article; if `or` then an article will be returned if it mentions any of the provided keywords. Available values: `and`, `or` Default value: `and` #### `keywordSearchMode` string Determines how the keywords provided with the `keyword` parameter are matched against the content. Use `phrase` (default) to match the keyword as an exact phrase, `exact` to use the Boolean query language (with `AND`, `OR`, `NOT`, `NEAR` and `NEXT` operators), or `simple` for a relevance-based search that does not require all terms to be present. See the [keyword search modes](/documentation?tab=keyword_search_options) section for details and examples. Available values: `phrase`, `exact`, `simple` Default value: `phrase` #### `conceptOper` string If more concepts are provided with the `conceptUri` parameter, what should be the Boolean operator used. If `and` (default) then all of the specified concepts have to be present in the article; if `or` then an article will be returned if it mentions any of the provided concepts. Available values: `and`, `or` Default value: `and` #### `categoryOper` string If more categories are provided with the `categoryUri parameter, what should be the Boolean operator used. If and then all of the specified categories have to be present in the article; if or then an article will be returned if it mentions any of the provided categories.` Available values: `and`, `or` Default value: `or` #### `startSourceRankPercentile` integer Starting [ranking percentile of the sources](https://github.com/EventRegistry/event-registry-python/wiki/Source-filtering#filtering-of-sources-based-on-their-ranking) to consider in the results (default: 0). Value should be in range 0-90 and divisible by 10. Default value: `0` #### `endSourceRankPercentile` integer Ending [ranking percentile of the sources](https://github.com/EventRegistry/event-registry-python/wiki/Source-filtering#filtering-of-sources-based-on-their-ranking) to consider in the results (default: 100). Value should be in range 10-100 and divisible by 10. Default value: `100` #### `minSentiment` integer The minimum value of the sentiment, the article should have. Valid value is any floating number between -1 (very negative) to 1 (very positive). 0 represents neutral sentiment. Note that setting the value will automatically reduce results to just English articles, since the sentiment can only be computed for English language. #### `maxSentiment` integer The maximum value of the sentiment, the article should have. Valid value is any floating number between -1 (very negative) to 1 (very positive). 0 represents neutral sentiment. Note that setting the value will automatically reduce results to just English articles, since the sentiment can only be computed for English language. #### `isDuplicateFilter` string Some articles can be duplicates of other articles. What should be done with them. Available values: `skipDuplicates`, `keepOnlyDuplicates`, `keepAll` Default value: `keepAll` #### `eventFilter` string Some articles describe a known event and some don't. This filter allows you to filter the resulting articles based on this criteria. Available values: `skipArticlesWithoutEvent`, `keepOnlyArticlesWithoutEvent`, `keepAll` Default value: `keepAll` #### `Returned Details` The API call can return more or less meta-data information about the articles, concepts, categories, sources and locations. Click to see the available properties that can be specified to determine the amount of returned information. #### `includeArticleTitle` boolean Set this parameter to true to include the article title in the response. Default value: `true` #### `includeArticleBasicInfo` boolean Set this parameter to true to include the core article information in the response. Default value: `true` #### `includeArticleBody` boolean Set this parameter to true to include the article body in the response. Default value: `true` #### `includeArticleEventUri` boolean Set this parameter to true to include the uri of the event (to which the article belongs) in the response. Default value: `true` #### `includeArticleSocialScore` boolean Set this parameter to true to include the information about how many times the article was shared on different social media. Default value: `false` #### `includeArticleSentiment` boolean Set this parameter to true to include the article sentiment in the response (value will be non-null only for English articles). Default value: `true` #### `includeArticleConcepts` boolean Set this parameter to true to include the article concepts in the response. Default value: `false` #### `includeArticleCategories` boolean Set this parameter to true to include the article categories in the response. Default value: `false` #### `includeArticleLocation` boolean Set this parameter to true to include the article location in the response. Default value: `false` #### `includeArticleImage` boolean Set this parameter to true to include the article image in the response. Default value: `true` #### `includeArticleAuthors` boolean Set this parameter to true to include the article authors in the response. Default value: `true` #### `includeArticleVideos` boolean Set this parameter to true to include the article videos in the response. Default value: `false` #### `includeArticleLinks` boolean Set this parameter to true to include the article links in the response. Default value: `false` #### `includeArticleExtractedDates` boolean Set this parameter to true to include article extracted dates in the response. Default value: `false` #### `includeArticleDuplicateList` boolean Set this parameter to true to include the list of duplicate articles in the response. Default value: `false` #### `includeArticleOriginalArticle` boolean Set this parameter to true to include the original article in the response. Default value: `false` #### `includeSourceTitle` boolean Set this parameter to true to include the source title in the response. Default value: `true` #### `includeSourceDescription` boolean Set this parameter to true to include the source description in the response. Default value: `false` #### `includeSourceLocation` boolean Set this parameter to true to include the source location in the response. Default value: `false` #### `includeSourceRanking` boolean Set this parameter to true to include the source ranking in the response. Default value: `false` #### `includeConceptLabel` boolean Set this parameter to true to include the concept label in the response. Default value: `true` #### `includeConceptImage` boolean Set this parameter to true to include the concept image in the response. Default value: `false` #### `includeConceptSynonyms` boolean Set this parameter to true to include the concept synonyms in the response. Default value: `false` #### `conceptLang` string Define the language of the concept label. Default value: `eng` #### `includeCategoryParentUri` boolean Set this parameter to true to include category parent uri in the response. Default value: `false` #### `includeLocationGeoLocation` boolean Set to true to include the geo location (latitude, longitude) for items that are locations. Default value: `false` #### `includeLocationPopulation` boolean Set to true to include the population size of the location. Default value: `false` #### `includeLocationGeoNamesId` boolean Set to true to include the GeoNames id of the location. Default value: `false` #### `includeLocationCountryArea` boolean Set to true to include the are of the location in squared km. Default value: `false` #### `includeLocationCountryContinent` boolean Set to true to get the continent of the country for location objects. Default value: `false` ## Request examples ### Example 1 Retrieve a stream of articles that were published in the last minute. Request body: ```json { "articleBodyLen": -1, "includeArticleConcepts": true, "includeArticleCategories": true, "apiKey": "YOUR_API_KEY" } ``` ```python from eventregistry import * er = EventRegistry(apiKey = YOUR_API_KEY) recentQ = GetRecentArticles(er, returnInfo = ReturnInfo(ArticleInfoFlags(concepts = True, categories = True))) starttime = time.time() while True: articleList = recentQ.getUpdates() print("%d articles were added since the last call" % len(articleList)) for article in articleList: print(article) print("sleeping for 60 seconds...") time.sleep(60.0 - ((time.time() - starttime) % 60.0)) ``` ```javascript var er = new erBase.EventRegistry(); var articleInfo = new erBase.ArticleInfoFlags({concepts: true, categories: true}); var returnInfo = new erBase.ReturnInfo({ articleInfo }); var recentQ = new erBase.GetRecentArticles(er, { returnInfo }); function fetchUpdates() { return recentQ.getUpdates().then((articleList) => { for (var i = 0; i < articleList.length; i++) { console.info(`Added article ${articleList[i]["uri"]}: ${articleList[i]["title"]}`); } return sleep(60 * 1000); }).then(() => { return fetchUpdates(); }); } fetchUpdates(); ``` ```typescript const er = new EventRegistry(); const articleInfo = new ArticleInfoFlags({bodyLen: -1, concepts: true, categories: true}); const returnInfo = new ReturnInfo({ articleInfo }); const recentQ = new GetRecentArticles(er, { returnInfo }); async function fetchUpdates() { const articleList = await recentQ.getUpdates(); for (let i = 0; i < articleList.length; i++) { console.info(`Added article ${articleList[i]["uri"]}: ${articleList[i]["title"]}`); } await sleep(60 * 1000); fetchUpdates(); } fetchUpdates(); ``` ### Example 2 Retrieve a stream of news articles that were published in the last 10 minutes that match the search criteria. The returned articles will be in English, German or Spanish language, and will either mention keyword "Tesla" or phrase "Elon Musk" in the title. Since it's not clear how many results you can expect in the 10 minutes we also increase the maximum number of articles returned (note that the number of used tokens will increase for each 100 returned articles). Request body: ```json { "recentActivityArticlesUpdatesAfterMinsAgo": 10, "recentActivityArticlesMaxArticleCount": 2000, "lang": [ "eng", "deu", "spa" ], "keyword": [ "Tesla", "Elon Musk" ], "keywordOper": "or", "keywordLoc": "title", "dataType": [ "news", "blog" ], "articleBodyLen": -1, "includeArticleConcepts": true, "includeArticleCategories": true, "apiKey": "YOUR_API_KEY" } ``` ```python recentQ = GetRecentArticles(er, returnInfo=ReturnInfo(ArticleInfoFlags(bodyLen=-1, concepts=True, categories=True)), recentActivityArticlesUpdatesAfterMinsAgo = 10, recentActivityArticlesMaxArticleCount = 2000, # use any filter from the Rest API list of valid parameters lang = ["eng", "deu", "spa"], keyword = ["Tesla", "Elon Musk"], keywordOper = "or", keywordLoc = "title", dataType = ["news", "blog"]) articleList = recentQ.getUpdates() for article in articleList: print(article) ``` ```javascript var er = new erBase.EventRegistry(); var articleInfo = new erBase.ArticleInfoFlags({bodyLen: -1, concepts: true, categories: true}); var returnInfo = new erBase.ReturnInfo({ articleInfo }); var params = { returnInfo: returnInfo, recentActivityArticlesUpdatesAfterMinsAgo: 10, recentActivityArticlesMaxArticleCount: 2000, lang: ["eng", "deu", "spa"], keyword: ["Tesla", "Elon Musk"], keywordOper: "or", keywordLoc: "title", dataType: ["news", "blog"], articleBodyLen: -1, includeArticleConcepts: true, includeArticleCategories: true, } var recentQ = new erBase.GetRecentArticles(er, { returnInfo }); recentQ.getUpdates().then((articleList) => { for (var i = 0; i < articleList.length; i++) { console.log(articleList[i]); } }); ``` ```typescript const er = new EventRegistry(); const articleInfo = new ArticleInfoFlags({bodyLen: -1, concepts: true, categories: true}); const returnInfo = new ReturnInfo({ articleInfo }); const params = { returnInfo: returnInfo, recentActivityArticlesUpdatesAfterMinsAgo: 10, recentActivityArticlesMaxArticleCount: 2000, lang: ["eng", "deu", "spa"], keyword: ["Tesla", "Elon Musk"], keywordOper: "or", keywordLoc: "title", dataType: ["news", "blog"], articleBodyLen: -1, includeArticleConcepts: true, includeArticleCategories: true, } const recentQ = new GetRecentArticles(er, { returnInfo }); recentQ.getUpdates().then((articleList) => { for (const article of articleList) { console.log(article); } }); ``` ### Example 3 To ensure that you will retrieve any matching articles that were published after your last API call, use the `recentActivityArticles.newestUri.blog` and `recentActivityArticles.newestUri.news` values when making the next API call. As you can see, we have specified the `recentActivityArticles.newestUri.blog` value as the `recentActivityArticlesBlogUpdatesAfterUri` parameter, and `recentActivityArticles.newestUri.news` value as the `recentActivityArticlesNewsUpdatesAfterUri` parameter. Instead of limiting the results based on the time when they were imported, they will be limited based on the last article(s) that were imported. Request body: ```json { "recentActivityArticlesNewsUpdatesAfterUri": "1307141581", "recentActivityArticlesBlogUpdatesAfterUri": "1235393368", "recentActivityArticlesMaxArticleCount": 2000, "lang": [ "eng", "deu", "spa" ], "keyword": [ "Tesla", "Elon Musk" ], "keywordOper": "or", "keywordLoc": "title", "dataType": [ "news", "blog" ], "articleBodyLen": -1, "includeArticleConcepts": true, "includeArticleCategories": true, "apiKey": "YOUR_API_KEY" } ``` ## Example output ```json { "recentActivityArticles": { "newestUpdate": "2023-05-16T10:55:42.874", "oldestUpdate": "2023-05-16T10:55:40.858", "newestUri": { "news": "7546857244" }, "currTime": "2023-05-16T10:56:15.342", "activity": [ { "uri": "7546857244", "lang": "eng", "isDuplicate": false, "date": "2023-05-16", "time": "10:55:00", "dateTime": "2023-05-16T10:55:00Z", "dateTimePub": "2023-05-16T10:54:00Z", "dataType": "news", "sim": 0, "url": "https://mondovisione.com/media-and-resources/news/sucden-financial-reports-robust-financial-results-for-2022/", "title": "Sucden Financial Reports Robust Financial Results For 2022", "body": "Sucden Financial, the multi-asset execution, clearing and liquidity provider, (trimmed)...", "source": { "uri": "mondovisione.com", "dataType": "news", "title": "Mondo Visione", "location": { "type": "place", "label": { "eng": "London" }, "lat": 51.50853, "long": -0.12574, "country": { "type": "country", "label": { "eng": "United Kingdom" }, "lat": 54.75844, "long": -2.69531 } }, "locationValidated": false }, "authors": [], "image": null, "eventUri": null, "location": null, "sentiment": 0.2784313725490195 }, { "#META ITEM": "other articles..." } ] } } ``` ## Output schema ```json { "type": "object", "properties": { "recentActivityArticles": { "type": "object", "properties": { "currTime": { "type": "string" }, "activity": { "type": "array", "items": { "type": "object", "properties": { "uri": { "type": "string" }, "lang": { "type": "string" }, "isDuplicate": { "type": "boolean" }, "date": { "type": "string" }, "time": { "type": "string" }, "dateTime": { "type": "string" }, "sim": { "type": "double" }, "url": { "type": "string" }, "title": { "type": "string" }, "body": { "type": "string" }, "source": { "type": "object", "properties": { "uri": { "type": "string" }, "dataType": { "type": "string" }, "title": { "type": "string" }, "description": { "type": "string" }, "location": { "type": "object", "properties": { "type": { "type": "string" }, "label": { "type": "object", "properties": { "eng": { "type": "string" } } } } }, "locationValidated": { "type": "boolean" }, "ranking": { "type": "object", "properties": { "importanceRank": { "type": "integer" } } } } }, "concepts": { "type": "array", "items": { "type": "object", "properties": { "uri": { "type": "string" }, "type": { "type": "string" }, "score": { "type": "integer" }, "label": { "type": "object", "properties": { "eng": { "type": "string" } } }, "image": { "type": "string" }, "synonyms": { "type": "object", "properties": {} }, "trendingScore": { "type": "object", "properties": { "news": { "type": "object", "properties": { "score": { "type": "double" }, "testPopFq": { "type": "integer" }, "nullPopFq": { "type": "integer" } } } } }, "location": { "type": "object", "properties": { "type": { "type": "string" }, "label": { "type": "object", "properties": { "eng": { "type": "string" } } } } } } } }, "categories": { "type": "array", "items": { "type": "object", "properties": { "uri": { "type": "string" }, "label": { "type": "string" }, "wgt": { "type": "integer" } } } }, "links": { "type": "array", "items": { "type": "string" } }, "videos": { "type": "array", "items": { "type": "string" } }, "image": { "type": "string" }, "duplicateList": { "type": "array", "items": { "type": "string" } }, "originalArticle": { "type": "string" }, "eventUri": { "type": "string" }, "location": { "type": "object", "properties": { "type": { "type": "string" }, "label": { "type": "object", "properties": { "eng": { "type": "string" } } } } }, "extractedDates": { "type": "array", "items": { "type": "object", "properties": { "amb": { "type": "boolean" }, "imp": { "type": "boolean" }, "date": { "type": "string" }, "textStart": { "type": "integer" }, "textEnd": { "type": "integer" } } } }, "shares": { "type": "object", "properties": { "facebook": { "type": "integer" } } }, "wgt": { "type": "integer" } } } } } } } } ```