Commit Graph

82 Commits

Author SHA1 Message Date
90e739a56e
Features/get route updates (#37)
* cleaned up get routes

* added a func to query articles by page

* removed unused imports

* merged page with listArticles as a optional
2023-01-31 08:19:23 -08:00
ada453e08a
Features/delete source and first dto (#36)
* updated db, added dto for ListSources, and added delete source

* updated from model > models

* updated to models

* sources now sends back a standard message

* updated subscription routes to have beter logid and swagger details

* moved the dto objects back to modles given they are not bound to the database

* cleaned up how we return the error

* cleaned up swag and updated models to take from the base apistatusmodel. less human errors this way

* cleaned up swag and updated models

* swag updated

* updated queue to return a router and also renamed it as it will hold all queue info later on

* removed config tag

* added subscription details route

* article routes have been moved to support dto

* updated discordwebhooks to use dto

* updated discordwebhookqueue to return details on the items via dto

* removed the example routes

* updated sources to use dto

* subscriptions moved to dto

* generated swag
2023-01-22 10:12:55 -08:00
a2147294d5
updated db, added dto for ListSources, and added delete source (#35) 2022-12-07 22:48:31 -08:00
ff4075383a
Features/subscription/features for portal (#34)
* added a route to delete subscriptions based on the ID given

* added a new route to find a record based on the name and source

* added a route to query Discord Web Hooks by Server and Channel names

* tested the endpoints and they seem good to test more

* updated some routes for subscriptions and formatted files

* removed debug file

* fixing some panic calls

* swag
2022-12-04 08:49:17 -08:00
c161658487
New endpoints for the portal to use (#31)
* added a route to delete subscriptions based on the ID given

* added a new route to find a record based on the name and source

* added a route to query Discord Web Hooks by Server and Channel names

* tested the endpoints and they seem good to test more
2022-11-30 21:43:53 -08:00
94da578c82
replaced sql command to order the new articles by source (#30) 2022-11-06 19:07:44 -08:00
337151b39d
updated articles route to order by pubdate and updated swagger (#29) 2022-11-06 17:17:46 -08:00
63b19452e5
removing build from merge. Only needed for PR (#28) 2022-08-21 20:06:01 -07:00
ac8eeaea01
Added discord management that was missing (#27) 2022-08-21 20:02:45 -07:00
b082c904ea
updated swagger for the new twitch change (#26) 2022-08-21 11:24:08 -07:00
3be2b52688
updated new twitch given it no longer needs a url param (#25) 2022-08-21 11:18:28 -07:00
06e50f956a
removed custom tags on source creation. Generates errors that we dont need currently (#19) 2022-07-27 10:56:44 -07:00
206251a3d1
added a new endpoint to get a list of sources by the source value (#18) 2022-07-26 13:27:34 -07:00
66c74802d9
Find installed browser for Docker (#17)
* trying a different method to have rod find to brower on disk for docker

* updated rod to find the browser on disk.  This is for docker support
2022-07-14 14:27:40 -07:00
9d662936d5
added chromium package to docker (#16) 2022-07-14 11:42:55 -07:00
79f3383fdc
Feature/dynamic swagger uri (#15)
* made the swagger uri based on env

* updated swagger to reflect the dynamic uri

* updated extamples
2022-07-14 09:59:55 -07:00
f01a8ff592
Features/swagger update (#14)
* ArticlesByTag added

* format cleanup
2022-07-13 21:31:53 -07:00
65f4281f92
Feature/deployment update (#13)
* updated to go 1.18.4 and added alpine dep.  Image tested locally

* added the first example of how to run the app

* added deployment notes

* updated bootup logic around the env and how to handle things

* swagger update
2022-07-13 15:52:25 -07:00
a1324ee1c1
Features/output discord (#12)
* basic output looks to be working

* cron was updated to add to the queue and post messages

* new route to make discord webhook subscriptions

* updated swag tags

* swag

* Updated delete subscription call

* removed the time value as it throws off the msg template

* updated logging

* updated swagger

* updated new subscription route

* Updated logging and remove items from the queue if they dont have a subscription

* updated getArticles to return the 50 newest for the portal

* added endpoint to see if an item exists already

* formatting

* updated listArticles

* added colors and updated the image

* Updated to use the pointer in twitch

* added the twitch login command to cron... it works now

* found a better way to disable http2 for reddit. Test worked right away too

* updated the cron tasks to run collected once and hour or longer depending on the service
2022-07-12 15:28:31 -07:00
0e0058506a
Feature Flags (#11)
* added feature flags around background workers

* background workers are moved to a new package as outputs are starting to get added

* package name was updated

* updated refs to the new input package

* query and sql updates on routes

* moved the services and starting to add discord web hook

* query update
2022-06-30 14:54:58 -07:00
713205bb03
Basic routes have been added (#10)
* basic routes are working with db context

* swagger is working along with swag gen

* cron was updated with a class and better db context, untested though

* sourcelist command added

* lost the pg package but added it back

* Updated the api startup for cron and api

* updated source routes and started to add article routes

* Updated cron add func calls

* updated swagger

* keeping articles basic for now as I dont need to pull them in yet

* swagger update

* added getarticlesbysourceid call

* adding the subscriptions table to track who to send notifications and where

* removed legacy columns from discordwebhooks that are no longer needed.

* added discord webhook routes

* updated routes

* Minor change to schema

* Updated routes to support subscriptions

* ignore .vscode
2022-06-19 22:02:44 -07:00
77d4fdf01a
version bump (#9) 2022-06-14 10:26:05 -07:00
75b66dd625
Feature/sql (#8)
* Using sqlc to generate queries and goose for migrations. The intial tests look good.

* moving the old calls away for now.  Might use this in a package later on.

* Added postgres driver

* Updated the dockerfile to support sql migrations

* added sqlc config file

* updated schema and starting a seed script

* updated models to use the database ones

* updated reddit cron to talk to the db

* added env for sql connection string

* got the reddit source working with the db and posting articles

* added sql packages

* added rule to ignore dev sql file

* added migration down statement for rolling back

* updated cron for reddit and youtube

* Updated reddit to follow a new standard pattern

* updated youtube to follow new patterns

* updated tests and brought them to the standard

* updated the seed migration

* all cron tasks should feed the db now

* updated app init

* bumped docker to 1.18.3

* disabled remote tests given secrets and lack of interfaces currently to run tests
2022-06-08 21:17:08 -07:00
333a4f5345
Features/twitch (#7)
* ran go mod tity

* updated formatting and moved to using some common errors

* some common source errors

* bumped go version

* a bulk of the work has been completed to parse twitch videos/users

* Added twitch config values

* added common errors

* moved the scheduler to its own package to get it out of main and also start adding tests

* added a new err for invalid author images

* Updated an error used

* tests updated to avoid name duplication

* new func to replace the cached source recor and updated getcontent to return collected posts

* updated scheduler ref

* moved to services/cron
2022-05-15 21:48:23 -07:00
11892b9a7b
Features/ffxiv (#6)
* starting the ffxiv reader

* working on getting the standard interface for sources based on the work for ffxiv

* got more of ffxiv working and updated tests

* Author and Description can be extracted and validated with tests

* added uuid package

* ffxiv core logic is working and testes updated to reflect it.

* Updated the scheduler with the current sources and moved them from main

* updated reddit to allow modern go to talk to the endpoint with a debug flag

* gave the func a better name

* cleaned up main

* Moved cache to its own package and updated tests"

* moved config to its own package and added basic tests

* updated imports

* minor update"

* interface update and cache model update

* updated the scheduler for basic services.  No DB calls yet

* updated db calls

* bypassed the reddit test as its flaky in github
2022-04-29 13:02:25 -07:00
eba63c27ef
Youtube source is now working (#5)
* added extra packages to help with parsing

* getting the core built for Youtube collection.  The feed can be pulled and starting to build the article object

* added some tests, reddit will need more love but youtube is starting off with more tests.  Starting to add Rod to pull missing values from the site

* Added rod to work with browser automation

* Moved the config inside the client as they can change within runtime and need to be refreshed on each client creation

* added more features to collect data per video and tests to support them.

* adding the cache and tests

* moved errors to var values at the top

* youtube is now getting collected and tests have been setup

* resolved an issue with the config

* setting up the first interface, its not used yet

* more updates to the cache service. Not finished yet and could see rework

* added logic to monitor Youtube.  Still basic logic that needs to be wired up to the database
2022-04-17 07:25:49 -07:00
e61870ec60
Create go-build.yml (#4) 2022-04-13 17:01:11 -07:00
4a947d89f2
removed unused import for build (#3) 2022-04-07 14:59:24 -07:00
2b959e140c
Features/reddit and dto (#2)
* updated db calls to new endpoints.  Not all are finished yet

* updated models to use the new dto objects to hide orm values

* updated reddit to build articles based on what type of post it is

* getting reddit ready to post the articles to the db

* Now able to post to the db under the new DTO object

* moved the reddit model to articles model to the struct and out of main
2022-04-07 14:53:40 -07:00
281fcb2d8a
got docker building with the correct binary (#1) 2022-04-02 12:09:43 -07:00
d8e85b1bd7 Getting the basics setup for the collector. It can pull reddit json and almost post the db api 2022-04-02 12:05:32 -07:00
c6620e3c88
Initial commit 2022-04-02 10:01:57 -07:00