Page Comparison

...

/orgs/{org}/audit-log

Data Source	Description	GitHub API endpoint	Collector service name	Type	Devo table	Available from release
Collaborators	Information about collaborators.	`/repos/{owner}/{repo}/collaborators` Repositories - GitHub Docs `metadata:read`	`collaborators`	`repository`	`vcs.github.repository.collaborators`	`v1.0.0`
Commits	Commits made in the repository	`/repos/{owner}/{repo}/commits` Repositories - GitHub Docs `contents:read`	`commits`	`repository`	`vcs.github.repository.commits`	`v1.0.0`
Forks	Forks created in the repository	`/repos/{owner}/{repo}/forks` Repositories - GitHub Docs `metadata:read`	`forks`	`repository`	`vcs.github.repository.forks`	`v1.0.0`
Events	Information about the different events such as resource creations or deletions.	`/repos/{owner}/{repo}/events` Eventos - GitHub Docs `metadata:read`	`events`	`repository`	`vcs.github.repository.events`	`v1.0.0`
Issue comments	Comments made in every issue.	`/repos/{owner}/{repo}/comments` Issue comments - GitHub Docs `issues:read or` `pull_requests:read`	`issue_comments`	`repository`	`vcs.github.repository.issue_comments`	`v1.0.0`
Subscribers	Information about the different users subscribed to one repository.	`/repos/{owner}/{repo}/subscribers` Watching - GitHub Docs `metadata:read`	`subscribers`	`repository`	`vcs.github.repository.subscribers`	`v1.0.0`
Pull requests	Pull requests made in the repository.	`/repos/{owner}/{repo}/pulls` `/repos/{owner}/{repo}/pulls/{pull_number}/commits` Pulls - GitHub Docs `pull_requests:read`	`pull_requests`	`repository`	`vcs.github.repository.pull_requests` `vcs.github.repository.pull_request_commits`	`v1.0.0`
Subscriptions	Repositories you are subscribed.	`/repos/{owner}/{repo}/subscription` Activity - GitHub Docs `metadata:read`	`subscriptions`	`repository`	`vcs.github.repository.subscriptions`	`v1.0.0`
Releases	Information about releases made in the repository.	`/repos/{owner}/{repo}/releases` Repositories - GitHub Docs `contents:read`	`releases`	`repository`	`vcs.github.repository.releases`	`v1.0.0`
Stargazers	Information about users who start repositories making them favorites	`/repos/{owner}/{repo}/stargazers` Starring - GitHub Docs `metadata:read`	`stargazers`	`repository`	`vcs.github.repository.stargazers`	`v1.0.0`	Audit	Organization auditory events.
`audit`	`organization`	`vcs.github.organization.audit`	`v1.0.0`	SSO Authorizations	Single sign-on authorization.	`/orgs/{org}/credential-authorizations` Organizations - GitHub Docs `organization_administration:read`	`sso_authorizations`	`organization`	`vcs.github.organization.sso_authorizations`	`v1.0.0`
Webhooks	Organization created webhooks.	`/orgs/{org}/hooks` Organizations - GitHub Docs `admin:org_hook`	`webhooks`	`organization`	`vcs.github.organization.webhooks`	`v1.0.0`
Dependabot Alerts	GitHub sends Dependabot alerts when we detect that your repository uses a vulnerable dependency or malware.	`/repos/{owner}/{repo}/dependabot/alerts` Dependabot alerts - GitHub Docs `vulnerability_alerts:read`	`dependabot_alerts`	`repository`	`vcs.github.organization.dependabot_alerts`	`v2.0.0`
Dependabot Secrets	Lists all secrets available in an organization without revealing their encrypted values.	`/orgs/{org}/dependabot/secrets` Dependabot secrets - GitHub Docs `admin:org`	`dependabot`	`organization`	`vcs.github.organization.dependabot`	`v2.0.0`
Actions	GitHub Actions for a repository.	`/repos/{owner}/{repo}/actions/runs` Workflow runs - GitHub Docs `actions:read`	`actions`	`repository`	`vcs.github.repository.actions`	`v2.0.0`
CodeScan	Code scanning is a feature that you use to analyze the code in a GitHub repository to find security vulnerabilities and coding errors.	`/repos/{owner}/{repo}/code-scanning/alerts` Code Scanning - GitHub Docs `security_events:read`	`codescan`	`repository`	`vcs.github.repository.codescan`	`v2.0.0`
Enterprise Audit	Enterprise Auditory Events	`/enterprises/{enterprise}/audit-log` REST API endpoints for organizations - GitHub Docs `admin:enterprise` `read:audit_log` `read:enterprise`	`enterprise_audit`	`enterprise`	`vcs.github.enterprise.audit`	`v2.0.0`
Organization Audit	Organization Auditory events	`/orgs/{org}/audit-log` REST API endpoints for organizations - GitHub Docs `read:audit_log`	`organization_audit`	`organization`	`vcs.github.organization.audit`	`v3.0.0`

...

Expand

title	Action Service

This service lists all workflows that run for a repository in GitHub. All events of this service are ingested into the table vcs.github.repository.actions.

Verify data collections

Puller Output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> No changes have been made in saved state. Returning saved state: {'pulling_date_from_config': '1640995200.0', 'last_pulled_date': '1641104263.0', 'ids': [1645452345]}
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> GithubDataPullerActions(github,444,actions,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> The collector will start pulling data since 2022-01-02T06:17:43Z
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Total number of repositories: 2
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Tag: vcs.github.api.repository.actions
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.
...
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 317; Number of duplicated events filtered out: 11; Number of events generated and sent: 306; Average of events per second: 23.234.

Info
The `@devo_pulling_id` value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that `Pull` action in Loxcope.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle Number of requests made: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40; Average of events per second: 23.234.

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Audit Service

This service gets the audit log (a sequence of activities) for an organization in GitHub. This service starts collecting 90 days back from the moment the persistence is reset. All events of this service are ingested into table vcs.github.organization.audit.

Info
This service generates a huge amount of events and it takes a lot of time and requests for the API to be up-to-date with this service.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> GithubDataPullerAudit(github,444,audit,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Tag: vcs.github.api.organization.audit
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 14.773.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 0; Number of events generated and sent: 60; Average of events per second: 14.709.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 90; Number of duplicated events filtered out: 0; Number of events generated and sent: 90; Average of events per second: 14.685.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.
...

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message. However, it takes a lot of time to reach the end of this service, as it generates a huge amount of events and starts pulling 90 days back:

Code Block

INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Statistics for this pull cycle Number of requests made: 10000; Number of events received: 300000; Number of duplicated events filtered out: 0; Number of events generated and sent: 300000; Average of events per second: 14.865.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after downloading a page when the pagination is required to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistance_reset_date parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Codescan Service

Code scanning is a feature that you can use to analyze the code in a GitHub repository to find security vulnerabilities and coding errors. This service returns the codescan results for each repository in case it is enabled.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'codescan': {}}
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> GithubDataPullerCodescan(github,444,codescan,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Get Codescan function called
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-1 -> 43
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-1 -> 9443322
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-2 -> 367
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-2 -> 4567887
....

After successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Statistics for this pull cycle Number of requests made: 120; Number of events received: 932; Number of duplicated events filtered out: 0; Number of events generated and sent: 932; Average of events per second: 23.593.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Collaborators Service

This service gets a list of collaborators for each repository in GitHub.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'codescan': {}}
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> GithubDataPullerCodescan(github,444,codescan,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Get Codescan function called
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-1 -> 43
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-1 -> 9443322
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-2 -> 367
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-2 -> 4567887
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Statistics for this pull cycle Number of requests made: 120; Number of events received: 932; Number of duplicated events filtered out: 0; Number of events generated and sent: 932; Average of events per second: 23.593.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Commits Service

This service gets a list of collaborators for each repository in GitHub.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'codescan': {}}
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> GithubDataPullerCodescan(github,444,codescan,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Get Codescan function called
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-1 -> 43
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-1 -> 9443322
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-2 -> 367
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-2 -> 4567887
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Statistics for this pull cycle Number of requests made: 120; Number of events received: 932; Number of duplicated events filtered out: 0; Number of events generated and sent: 932; Average of events per second: 23.593.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Dependabot

This service lists all secrets available in an organization without revealing their encrypted values. All events of this service are ingested into table vcs.github.repository.dependabot.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> State saved: {'old_persistence_reset_date': 'test1', 'dependabot': {}}
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> GithubDataPullerDependabot(github,444,dependabot,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> No more pages have been detected ahead for org my-organization
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> New items found for org my-organization -> 4
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Persistence saved for org my-organization -> 46463737382
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Statistics for this pull cycle Number of requests made: 1; Number of events received: 4; Number of duplicated events filtered out: 0; Number of events generated and sent: 4; Average of events per second: 5.743.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Dependabot Alerts

This service returns the Dependabot Alerts for each repository. GitHub generates an alert when a repository uses a vulnerable dependency or malware. All events of this service are ingested into table vcs.github.repository.dependabot_alerts.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'dependabot_alerts': {}}
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Get Dependabot Alerts function called
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> New items found for repo repo-1 -> 12
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Persistence saved for repo-1 -> 94445
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> New items found for repo repo-2 -> 3
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Persistence saved for repo-2 -> 45556
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 122; Number of duplicated events filtered out: 0; Number of events generated and sent: 122; Average of events per second: 13.63.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Events

This service returns the events for each repository. All events of this service are ingested into table vcs.github.repository.events.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> State saved: {'old_persistence_reset_date': 'prueba-2', 'events': {}}
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> GithubDataPullerEvents(github,444,events,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Get Events function called
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Partial statistics: Pages retrieved 20, items buffered 600
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> New items found for repo repo-1 -> 740
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Persistence saved for repo-1 -> 456789
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> New items found for repo repo-2 -> 356
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Persistence saved for repo-2 -> 5678567
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Statistics for this pull cycle Number of requests made: 89; Number of events received: 1562; Number of duplicated events filtered out: 0; Number of events generated and sent: 1562; Average of events per second: 79.13.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Forks

This service returns the forks for each repository. All events of this service are ingested into table vcs.github.repository.forks.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> State saved: {'old_persistence_reset_date': 'prueba-3', 'forks': {}}
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> GithubDataPullerForks(github,444,forks,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Get Forks function called
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> New items found for repo repo-1 -> 3
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Persistence saved for repo-1 -> 9623469344
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> New items found for repo repo-2 -> 2
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Persistence saved for repo-2 -> 5678234564
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 56; Number of duplicated events filtered out: 0; Number of events generated and sent: 56; Average of events per second: 12.128.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Issue Comments Service

This service gets the list of issue comments made into a repository in GitHub. All events of this service are ingested into table vcs.github.repository.issue_comments.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> No changes have been made in saved state. Returning saved state: {'pulling_date_from_config': '1640995200.0', 'last_pulled_date': '1641852073.0', 'ids': [1009384492]}
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> The collector will start pulling data since 2022-01-10T22:01:13Z
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Total number of repositories: 2
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Tag: vcs.github.api.repository.issue_comments
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 60; Number of events generated and sent: 0; Average of events per second: 0.000.
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 105; Number of events generated and sent: 15; Average of events per second: 1.307.
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 6; Number of events received: 180; Number of duplicated events filtered out: 160; Number of events generated and sent: 20; Average of events per second: 1.294.
...

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Statistics for this pull cycle Number of requests made: 6; Number of events received: 180; Number of duplicated events filtered out: 160; Number of events generated and sent: 20; Average of events per second: 1.294.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle Number of requests made: 6; Number of events received: 180; Number of duplicated events filtered out: 160; Number of events generated and sent: 20; Average of events per second: 1.294.

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Pull Request Service

This service returns the pull request for each repository, and the associate commits for each pull request. All events of this service are ingested into table vcs.github.repository.pull_requests and vcs.github.repository.pull_requests.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> State saved: {'old_persistence_reset_date': '27-Oct-2022', 'pull_requests': {}, 'pull_request_commits': {}}
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 1
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> bdb806bd6218552c7b3b6507803e48694b5591b7
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 1
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> ccc00f42d56d59bcb375a327f163bb8b737f376d
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 17
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> 8f1af254f6fffad8718c7c68a7f67778bc6c6b3f
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 1
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> a0d5d609978be33b7b7c37b2ea2a400d6102ccda
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 8
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> e99dfc7b7075250ca685106bab18f31f7e62f3c5
...

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Statistics for this pull cycle Number of requests made: 172; Number of events received: 1333; Number of duplicated events filtered out: 0; Number of events generated and sent: 1333; Average of events per second: 15.183.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	Releases Service

This service returns the releases for each repository. All events of this service are ingested into table vcs.github.repository.releases.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> State saved: {'old_persistence_reset_date': '27-Oct', 'releases': {}}
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> GithubDataPullerReleases(github,444,releases,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Get Releases function called
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> New items found for repo repo-1 -> 0
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> New items found for repo repo-2 -> 0
....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.0.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	SSO Authorization Service

This service returns Single Sign On authorization for all organizations. All events of this service are ingested into table vcs.github.organizations.sso_authorizations.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-10-16T12:00:00Z', 'sso_authorizations': {}}
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> No more pages have been detected ahead for org my-organization
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> New items found for org my-organization -> 58
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Persistence saved for org my-organization -> 40487930
.....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Statistics for this pull cycle Number of requests made: 2; Number of events received: 58; Number of duplicated events filtered out: 0; Number of events generated and sent: 58; Average of events per second: 80.759.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	Stargazers

This service returns information about the users who starts each repository, making it a favorite. All events of this service are ingested into table vcs.github.repository.stargazers.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-11-01T12:34:21Z', 'stargazers': {}}
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> GithubDataPullerStargazers(github,444,stargazers,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-1 -> 9
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Persistence saved for repo-1 -> 1966093
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-2 -> 0
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-3
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-3 -> 0
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-4
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-4 -> 0
...

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 33; Number of duplicated events filtered out: 0; Number of events generated and sent: 33; Average of events per second: 3.049.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	Subscribers

This service returns information about users subscribed to one repository. All events of this service are ingested into table vcs.github.repository.subscribers.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-10-27', 'subscribers': {}}
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> GithubDataPullerSubscribers(github,444,subscribers,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> New items found for repo repo-1 -> 60
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence saved for repo-1 -> 1234236
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> New items found for repo repo-2 -> 54
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence saved for repo-2 -> 2342343
...
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> No more pages have been detected ahead for repo repo-N
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> New items found for repo repo-N -> 1
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence saved for repo-N -> 3453457
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Statistics for this pull cycle Number of requests made: 10; Number of events received: 178; Number of duplicated events filtered out: 0; Number of events generated and sent: 178; Average of events per second: 11.194.
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> The data is up to date!
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Data collection completed. Elapsed time: 15.904 seconds. Waiting for 44.096 second(s) until the next one

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Statistics for this pull cycle Number of requests made: 10; Number of events received: 178; Number of duplicated events filtered out: 0; Number of events generated and sent: 178; Average of events per second: 11.194.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	Subscription

This service returns information about the repositories the user (used to make the pulling) is subscribed to. All events of this service are ingested into table vcs.github.repository.subscriptions.

Info
Depending on the kind of user you are using to make the data pulling, it could make no sense for you checking if the user is subscribed to the Organization repositories.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-10-27', 'subscriptions': {}}
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Pull Started
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) ->  404 Did not found any watchers/subscriptions for the repository
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> New items found for repo repo-1 -> 0
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) ->  404 Did not found any watchers/subscriptions for the repository
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> New items found for repo repo-2 -> 0
...
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) ->  404 Did not found any watchers/subscriptions for the repository
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> No more pages have been detected ahead for repo repo-N
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> New items found for repo repo-N -> 0
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Statistics for this pull cycle Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> The data is up to date!
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Data collection completed. Elapsed time: 8.960 seconds. Waiting for 51.040 second(s) until the next one

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Statistics for this pull cycle Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	Webhooks

List of webhooks created by the organization. All events of this service are ingested into table vcs.github.organizations.webhooks.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> State saved: {'old_persistence_reset_date': '27-October', 'webhooks': {}}
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> GithubDataPullerWebhooks(github,444,webhooks,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> No more pages have been detected ahead for org my-organization
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> New items found for org my-organization -> 1
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Persistence saved for org my-organization -> 324567455
.....

After the successful execution of the collector, you should be able to see the following log message:

Code Block

INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Statistics for this pull cycle Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 5.679.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the Partial reference.

Partial statistics: Pages retrieved 10, items buffered 300

Restart the persistence

This service makes use of persistence. To restart the persistence, the persistence_reset_date parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.

Expand

title	Enterprise audit service

Description

This service gets the audit log (a sequence of activities) for an enterprise in GitHub.

Note
This service generates a huge amount of events and it takes a lot of time and requests to the API to be up-to-date with this service. Use the parameter `since` to set a near date as the beginning.

Devo categorization and destination

All events of this service are ingested into the table vcs.github.enterprise.audit.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> GithubDataPullerAudit(github,444,audit,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Tag: vcs.github.api.organization.audit
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 14.773.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 0; Number of events generated and sent: 60; Average of events per second: 14.709.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 90; Number of duplicated events filtered out: 0; Number of events generated and sent: 90; Average of events per second: 14.685.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.
...

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message. However, it takes a lot of time to reach the end of this service, as it generates a huge amount of events and starts pulling 90 days back:

Code Block

INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Statistics for this pull cycle Number of requests made: 10000; Number of events received: 300000; Number of duplicated events filtered out: 0; Number of events generated and sent: 300000; Average of events per second: 14.865.

Info
The `@devo_pulling_id` value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that `Pull` action in the Data Search area of Devo.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed in the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

Expand

title	Organization audit service

Description

This service gets the audit log (a sequence of activities) for an organization in Github.

Note
This service generates a huge amount of events and it takes a lot of time and requests to the API to be up-to-date with this service. Use the parameter `since` to set a near date as the beginning

Devo categorization and destination

All events of this service are ingested into table vcs.github.organization.audit

Verify data collection

Puller Output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

Code Block

INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> GithubDataPullerAudit(github,444,audit,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Tag: vcs.github.api.organization.audit
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 14.773.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 0; Number of events generated and sent: 60; Average of events per second: 14.709.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 90; Number of duplicated events filtered out: 0; Number of events generated and sent: 90; Average of events per second: 14.685.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.
...

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message. However, it takes a lot of time to reach the end of this service, as it generates a huge amount of events and starts pulling 90 days back:

Code Block

INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Statistics for this pull cycle Number of requests made: 10000; Number of events received: 300000; Number of duplicated events filtered out: 0; Number of events generated and sent: 300000; Average of events per second: 14.865.

Info
The `@devo_pulling_id` value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that `Pull` action in Loxcope.

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.

Restart the persistence

This service makes use of persistence. To restart the persistence, the since parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.

...

Page Comparison

Versions Compared

Old Version 24

New Version 25

Key

Verify data collections

Puller Output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Verify data collection

Puller output

Restart the persistence

Description

Devo categorization and destination

Verify data collection

Puller output

Restart the persistence

Description

Devo categorization and destination

Verify data collection

Puller Output

Restart the persistence