Commit Graph

6 Commits

Author SHA1 Message Date
James Tombleson 77d4fdf01a
version bump (#9) 2022-06-14 10:26:05 -07:00
James Tombleson 75b66dd625
Feature/sql (#8)
* Using sqlc to generate queries and goose for migrations. The intial tests look good.

* moving the old calls away for now.  Might use this in a package later on.

* Added postgres driver

* Updated the dockerfile to support sql migrations

* added sqlc config file

* updated schema and starting a seed script

* updated models to use the database ones

* updated reddit cron to talk to the db

* added env for sql connection string

* got the reddit source working with the db and posting articles

* added sql packages

* added rule to ignore dev sql file

* added migration down statement for rolling back

* updated cron for reddit and youtube

* Updated reddit to follow a new standard pattern

* updated youtube to follow new patterns

* updated tests and brought them to the standard

* updated the seed migration

* all cron tasks should feed the db now

* updated app init

* bumped docker to 1.18.3

* disabled remote tests given secrets and lack of interfaces currently to run tests
2022-06-08 21:17:08 -07:00
James Tombleson 333a4f5345
Features/twitch (#7)
* ran go mod tity

* updated formatting and moved to using some common errors

* some common source errors

* bumped go version

* a bulk of the work has been completed to parse twitch videos/users

* Added twitch config values

* added common errors

* moved the scheduler to its own package to get it out of main and also start adding tests

* added a new err for invalid author images

* Updated an error used

* tests updated to avoid name duplication

* new func to replace the cached source recor and updated getcontent to return collected posts

* updated scheduler ref

* moved to services/cron
2022-05-15 21:48:23 -07:00
James Tombleson 11892b9a7b
Features/ffxiv (#6)
* starting the ffxiv reader

* working on getting the standard interface for sources based on the work for ffxiv

* got more of ffxiv working and updated tests

* Author and Description can be extracted and validated with tests

* added uuid package

* ffxiv core logic is working and testes updated to reflect it.

* Updated the scheduler with the current sources and moved them from main

* updated reddit to allow modern go to talk to the endpoint with a debug flag

* gave the func a better name

* cleaned up main

* Moved cache to its own package and updated tests"

* moved config to its own package and added basic tests

* updated imports

* minor update"

* interface update and cache model update

* updated the scheduler for basic services.  No DB calls yet

* updated db calls

* bypassed the reddit test as its flaky in github
2022-04-29 13:02:25 -07:00
James Tombleson eba63c27ef
Youtube source is now working (#5)
* added extra packages to help with parsing

* getting the core built for Youtube collection.  The feed can be pulled and starting to build the article object

* added some tests, reddit will need more love but youtube is starting off with more tests.  Starting to add Rod to pull missing values from the site

* Added rod to work with browser automation

* Moved the config inside the client as they can change within runtime and need to be refreshed on each client creation

* added more features to collect data per video and tests to support them.

* adding the cache and tests

* moved errors to var values at the top

* youtube is now getting collected and tests have been setup

* resolved an issue with the config

* setting up the first interface, its not used yet

* more updates to the cache service. Not finished yet and could see rework

* added logic to monitor Youtube.  Still basic logic that needs to be wired up to the database
2022-04-17 07:25:49 -07:00
James Tombleson d8e85b1bd7 Getting the basics setup for the collector. It can pull reddit json and almost post the db api 2022-04-02 12:05:32 -07:00