Intermittent issue with devices returning Timeout (Device timeout waiting response)

[RafaelGomes01]

Good afternoon everyone,

I am using the Anviz framework to fetch time clock records from my clients and integrate them into our time clock platform. Our system is multi-tenant, meaning we serve multiple clients through the same API. To achieve this, we implemented a service that synchronizes data every 20 minutes during business hours and every hour outside of business hours.

Most of the time, the system works perfectly, keeping records up to date. However, occasionally, some active devices return the error Device timeout waiting response.. This issue persists for days and sometimes even weeks, preventing records from being updated. The problem occurs even when connectivity issues are ruled out since the system was operating correctly before the error occurred.

I would like to understand what might be causing this behavior and if there are any recommendations to resolve it.

Basic system flow:

The service runs the synchronization process at regular intervals, iterating through all tenants.
For each tenant, we connect to the Anviz devices and fetch records using the Anviz SDK.
For devices experiencing the mentioned error, the process halts, and records are not updated.
Below is the code for further context:

Data synchronization code:

private async Task<bool> SyncAnvizRecords()
{
    using (IServiceScope scope = _serviceProvider.CreateScope())
    {
        IServiceProvider serviceProvider = scope.ServiceProvider;
        TenantsDbContext _tenantService = serviceProvider.GetRequiredService<TenantsDbContext>();
        IMultiTenantContextAccessor _multiTenantContextAccessor = serviceProvider.GetRequiredService<IMultiTenantContextAccessor>();
        IMultiTenantStore<TenantInfo> tenantStore = serviceProvider.GetRequiredService<IMultiTenantStore<TenantInfo>>();
        List<TenantInfo> tenants = await _tenantService.TenantInfo.ToListAsync();

        if (tenants.Count > 0)
        {
            foreach (TenantInfo tenant in tenants)
            {
                try
                {
                    if (tenant.ConnectionString == null)
                        throw new InvalidOperationException($"Connection string not found for tenantId: {tenant.Name}");

                    DbContextOptionsBuilder<ApplicationDbContext> optionsBuilder = new DbContextOptionsBuilder<ApplicationDbContext>();
                    optionsBuilder.UseSqlServer(tenant.ConnectionString);

                    MultiTenantContext<TenantInfo> tenantContext = new MultiTenantContext<TenantInfo> { TenantInfo = tenant };
                    _multiTenantContextAccessor.MultiTenantContext = tenantContext;

                    using (ApplicationDbContext dbContext = new ApplicationDbContext(optionsBuilder.Options, _multiTenantContextAccessor, _config, serviceProvider.GetRequiredService<IHttpContextAccessor>()))
                    {
                        using (var scopeSecond = _serviceProvider.CreateScope())
                        {
                            var scopedProvider = scopeSecond.ServiceProvider;

                            var service = scopedProvider.GetRequiredService<IAnvizRecordService>();
                            await service.GetRecordV3();
                        }
                    }
                }
                catch (Exception ex)
                {
                    Console.Error.WriteLine($"[Erro Detalhado] Tenant: {tenant.Name} - {DateTime.Now}");
                    Console.Error.WriteLine($"Mensagem: {ex.Message}");
                    Console.Error.WriteLine("‎");
                }
            }
        }
    }

    return true;
}

Record-fetching code:

public async Task<bool> GetRecordV3()
{
    List<AnvizEntity> anviz = await _context.Anviz.Where(Anviz => Anviz.Active && Anviz.Status).ToListAsync();

    foreach (AnvizEntity device in anviz)
    {
        TcpClient relogio = ConnectToAnvizDevice(device.Ip, device.Port);
        AnvizDevice conexao = new AnvizDevice(relogio);

        if (device.Username != null && device.Password != null)
        {
            var manager = new AnvizManager();
            manager.AuthenticateConnection = true;

            AESHelper aesHelper = new AESHelper();
            string decryptedPassword = aesHelper.Decrypt(device.Password);
            _ = conexao.SetConnectionPassword(device.Username, decryptedPassword);
        }

        List<AnvizSdkRecord> sdkRecords = await conexao.DownloadRecords(true);

        foreach (AnvizSdkRecord anvizSdkRecord in sdkRecords.OrderBy(sdkRecord => sdkRecord.DateTime))
        {
            DateTime normalizedAnvizSdkRecordDateTime = anvizSdkRecord.DateTime.AddTicks(-(anvizSdkRecord.DateTime.Ticks % TimeSpan.TicksPerSecond));

            bool existedRecord = _context.AnvizRecords
                .Where(anvizRecord =>
                    anvizRecord.UserCode == anvizSdkRecord.UserCode &&
                    anvizRecord.DateTime == normalizedAnvizSdkRecordDateTime &&
                    anvizRecord.BackupCode == anvizSdkRecord.BackupCode &&
                    anvizRecord.RecordType == anvizSdkRecord.RecordType &&
                    anvizRecord.WorkType == anvizSdkRecord.WorkType)
                .Any();

            if (existedRecord)
                continue;

            AnvizRecord newAnvizRecord = new AnvizRecord(anvizSdkRecord.UserCode, anvizSdkRecord.DateTime, anvizSdkRecord.BackupCode, anvizSdkRecord.RecordType, anvizSdkRecord.WorkType);
            _context.AnvizRecords.Add(newAnvizRecord);

            await _recordService.AddRecordFromClock(newAnvizRecord);
        }
        conexao.Dispose();
    }

    _context.SaveChanges();
    return true;
}

If you need additional details or specific tests, I am available to assist in investigating this issue.

Thank you for your attention, and I look forward to your response.