Embase_Conference_Workflow_Scheduler/QUICK_START.md

Embase Conference Scheduler - Quick Start

Project Architecture Summary

Clean Architecture with 4 layers following Microsoft best practices:

├── Domain          (Entities, Interfaces, Config models) — No dependencies
├── Application     (Business logic, Orchestration) — Depends on Domain
├── Infrastructure  (Dapper/PostgreSQL, SFTP, ZIP) — Depends on Domain
└── Worker          (Quartz Job, DI, Program.cs) — Depends on all

Technology Stack

• .NET 8 Worker Service
• PostgreSQL with Dapper ORM
• Quartz.NET for scheduling
• Serilog for logging
• SSH.NET for SFTP
• Docker for containerization

---

Initial Setup

1. Database Setup

psql -d embase -f Database/create_tracking_table.sql

This creates tblEmbaseConferenceDispatch tracking table.

2. Configuration

Configure per environment in src/EmbaseConferenceScheduler.Worker/:

File	Purpose
appsettings.json	Common settings (Serilog, defaults)
appsettings.Development.json	Local dev overrides
appsettings.Staging.json	Staging environment
appsettings.Production.json	Production environment

Key settings:

• ConnectionStrings:EmbaseDb - PostgreSQL connection
• Sftp - SFTP server details
• Packaging - PDF paths, ZIP naming
• Scheduler:CronExpression - Job schedule

---

Running Locally

Development Mode

cd src/EmbaseConferenceScheduler.Worker
dotnet run --environment Development

Specific Environment

dotnet run --environment Staging
dotnet run --environment Production

Build Release

dotnet build -c Release

---

Docker Deployment

Using Layered Architecture Files

1. Configure appsettings files: All settings are in src/EmbaseConferenceScheduler.Worker/appsettings.{Environment}.json

   • Edit appsettings.Production.json for production
   • Edit appsettings.Staging.json for staging
   • Edit appsettings.Development.json for local development

2. Build Docker image:

   docker build -t embase-conference-scheduler:latest .
   

3. Run container:

   docker run -d \
    -e DOTNET_ENVIRONMENT=Production \
    -v /data/production/articles/pdf:/production/articles/pdf:ro \
    -v embase-logs:/logs \
    --name embase-conference-scheduler \
    embase-conference-scheduler:latest
   

4. View logs:

   docker logs -f embase-conference-scheduler
   

5. Stop:

   docker stop embase-conference-scheduler
   docker rm embase-conference-scheduler
   

---

Environment Variables (Docker Override)

Override any appsettings value via environment variables:

# Database
ConnectionStrings__EmbaseDb="Host=...;Database=...;Username=...;Password=..."

# SFTP
Sftp__Host="sftp.example.com"
Sftp__Username="user"
Sftp__Password="password"

# Scheduler
Scheduler__CronExpression="0 0 3 * * ?"

# Packaging
Packaging__PdfSourcePath="/custom/path"

---

Scheduler CRON

Default schedules:

Environment	CRON	Time
Development	0 */5 * * * ?	Every 5 minutes
Staging	0 0 3 * * ?	Daily 03:00 IST
Production	0 0 2 * * ?	Daily 02:00 IST

CRON Format: Seconds Minutes Hours Day Month DayOfWeek

---

Project Structure

src/
├── EmbaseConferenceScheduler.Domain/
│   ├── Entities/                    # Business entities
│   ├── Interfaces/                  # Repository & service contracts
│   └── Configuration/               # Settings models
│
├── EmbaseConferenceScheduler.Application/
│   └── Services/
│       └── PackagingService.cs      # Core business orchestration
│
├── EmbaseConferenceScheduler.Infrastructure/
│   ├── Persistence/
│   │   └── ConferenceAbstractRepository.cs  # Dapper + PostgreSQL
│   ├── FileTransfer/
│   │   └── SftpService.cs          # SSH.NET SFTP
│   └── FileOperations/
│       └── ZipService.cs            # ZIP creation
│
└── EmbaseConferenceScheduler.Worker/
    ├── Program.cs                   # Host bootstrap & DI
    ├── Jobs/
    │   └── ConferenceAbstractPackagingJob.cs  # Quartz job
    ├── Configuration/
    │   ├── DependencyInjection.cs   # Service registration
    │   └── QuartzConfiguration.cs   # Scheduler setup
    └── appsettings.*.json           # Environment configs

---

##Workflow Summary

1. Quartz triggers job daily (CRON schedule)
2. Query unprocessed conference abstracts from PostgreSQL
   • tbldiscardeditemreport WHERE lotid NOT IN tblEmbaseConferenceDispatch
3. Group articles by SourceId (one ZIP per source)
4. Copy PDFs to temp folder
5. Create ZIP with name emconflumXXXXXXX.zip
6. Upload to SFTP
7. Save dispatch records to prevent reprocessing

---

Troubleshooting

Check logs

# Docker
docker logs -f embase-conference-scheduler

# Local
# Logs written to /logs/scheduler.log or C:/dev/embase/logs/ (Dev)

Common issues

Job not running:

• Check CRON expression validity
• Verify Quartz started (look for "Scheduler started" in logs)

Database errors:

• Verify connection string
• Check database user permissions
• Ensure tracking table exists

SFTP failures:

• Test connectivity: telnet sftp-host 22
• Verify credentials
• Check private key file permissions (if using key auth)

PDFs not found:

• Verify Packaging:PdfSourcePath is correct
• Check file permissions

---

Development Tips

Test with immediate execution

Change CRON to trigger every minute:

"Scheduler": {
  "CronExpression": "0 * * * * ?"
}

Use development paths

Configure local Windows paths in appsettings.Development.json:

"Packaging": {
  "PdfSourcePath": "C:/dev/embase/pdfs/",
  "TempWorkingPath": "C:/dev/embase/tmp/"
}

Enable debug logging

"Serilog": {
  "MinimumLevel": {
    "Default": "Debug"
  }
}

---

Next Steps

• Configure production database connection
• Set up SFTP credentials/key
• Test with sample data
• Schedule production CRON
• Monitor first production run
• Set up alerts for failures

---

For detailed architecture documentation, see README_Architecture.md