Fix detection of some registers

[MabezDev]

The current output from master shows the failed parsing of some registers.

Failed to match reg info at esp-idf/components/soc/esp32/include/soc/uhci_reg.h:1007
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/i2c_reg.h:944
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/frc_timer_reg.h:34
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/frc_timer_reg.h:45
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/frc_timer_reg.h:48
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rtc_cntl_reg.h:1833
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:18
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:20
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:22
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:24
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:26
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:28
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:30
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:32
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:35
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:37
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:39
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:41
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:43
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:45
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:47
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:49
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:51
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:53
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:55
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:57
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:59
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:61
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:63
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:65
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/sdmmc_reg.h:67
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/hwcrypto_reg.h:27
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/hwcrypto_reg.h:29
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/hwcrypto_reg.h:31
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/hwcrypto_reg.h:33
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/hwcrypto_reg.h:35
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/spi_reg.h:135
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:18
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:20
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:22
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:24
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:26
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:28
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:30
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rmt_reg.h:32
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/host_reg.h:973
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rtc_i2c_reg.h:155
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rtc_i2c_reg.h:231
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/rtc_i2c_reg.h:237
Failed to match reg info at esp-idf/components/soc/esp32/include/soc/uart_reg.h:22

[jessebraham]

I haven't had a chance to dig into this too deep yet, but I think I may have found the source of the problem. For certain definitions, the transition from State::FindReg to State::FindBitFieldInfo seems to be consuming an additional line, and so re_reg_bit_info is not matching on the line we want it to.

I printed out a bit of debug info to show this. I've included a couple of examples below:

Failed to match reg info at esp-idf/components/soc/esp32/include/soc/uart_reg.h:22
Register {
    name: "UART_FIFO_AHB",
    address: 0,
    width: 0,
    description: "UART_FIFO_AHB",
    reset_value: 0,
    detailed_description: None,
    bit_fields: [],
}
PrevLine: #define UART_FIFO_AHB_REG(i)  (REG_UART_AHB_BASE(i) + 0x0)
Line:     #define UART_FIFO_REG(i)          (REG_UART_BASE(i) + 0x0)

Failed to match reg info at esp-idf/components/soc/esp32/include/soc/uhci_reg.h:1007
Register {
    name: "UHCI_ACK_NUM",
    address: 108,
    width: 0,
    description: "UHCI_ACK_NUM",
    reset_value: 0,
    detailed_description: None,
    bit_fields: [],
}
PrevLine: #define UHCI_ACK_NUM_REG(i)          (REG_UHCI_BASE(i) + 0x6C)
Line:

As you can see in both these cases, the previous line is the one we want to be matching on. The line number printed in the error message corresponds with the previous line.

[MabezDev]

Thanks for looking into this! I've just poked around in those files, and in those two instances, I think the parser it behaving normally.

The way idf2svd works is by finding defines that end in _REG as a starting point. Then the next line(s) will be bitfield information.

The full snippet from uart_reg.h looks like this

#define UART_FIFO_AHB_REG(i)  (REG_UART_AHB_BASE(i) + 0x0) // idf2svd expects a comment with bitfield info below here
#define UART_FIFO_REG(i)          (REG_UART_BASE(i) + 0x0) // and below here

/* UART_RXFIFO_RD_BYTE : RO ;bitpos:[7:0] ;default: 8'b0 ; */
/*description: This register stores one byte data  read by rx fifo.*/
#define UART_RXFIFO_RD_BYTE  0x000000FF
#define UART_RXFIFO_RD_BYTE_M  ((UART_RXFIFO_RD_BYTE_V)<<(UART_RXFIFO_RD_BYTE_S))
#define UART_RXFIFO_RD_BYTE_V  0xFF
#define UART_RXFIFO_RD_BYTE_S  0

I think this more of an issue with the consistency of the documentation of the idf; though with that in mind, what is the best course of action to take.

I only looked at uart and uhci so far but its possible that there are bugs in the parsing for other peripherals.

[MabezDev]

From page 351 in the reference manual UART_FIFO_REG is a valid reg with UART_RXFIFO_RD_BYTE as its only bitfield, it seems there has been a newline mistakenly added.

Ideally the parser should really be able to cope with this but as far as I can tell UART_FIFO_AHB_REG doesn't make sense in that position

[MabezDev]

In RTC_CNTL, the read only info uses a '0' instead of a 'O'.

/* RTC_CNTL_RDY_FOR_WAKEUP : R/0; bitpos:[19]; default: 0 */

should be

/* RTC_CNTL_RDY_FOR_WAKEUP : R/O; bitpos:[19]; default: 0 */

[jessebraham]

Gotcha, thanks for clearing that up. The output just seemed off to me initially.

The error The following registers failed to parse ["FRC_TIMER_LOAD", "FRC_TIMER_COUNT"] is printed as well. The lines these items are defined on both end with comments, which in turn end with ). The comments were being captured along with the offsets in my testing. I think we just need to lazily match on the last group to fix this:

r"\#define[\s*]+([^\s*]+)_REG\(i\)[\s*]+\(REG_([0-9A-Za-z_]+)_BASE[\s*]*\(i\) \+ (.*?)\)"

(Note the addition of ? in the last capture group)

[MabezDev]

Ah good spot! Would you mind PR'ing that change to the regex?

For the items where the docs are wrong (or missing), I guess we'll have to fix the docs in our own fork (unless you have any other ideas?) but I think its better to worry about that later.

[jessebraham]

I've submitted a PR to fix the register parsing issue. The items outlined in the first comment still need to be resolved at this point.

I'm not against maintaining our own fork, as that's probably easier and more reliable than trying to be clever and coding around the inconsistencies. But as you said that can probably be addressed at another time, it's documented here at least for the time being.

[bschwind]

Hi there, I recently commented on your reddit thread about helping out. Still reading through the code, but here are my assumptions so far, please correct me if I'm wrong:

• The SVD file is a standardized XML file which describes all peripherals for the chip, along with their registers, memory offsets, etc.
• This project is reading the defines in the esp-idf code, and generating an SVD file
• Currently the program outputs Failed to match reg info at... on registers it couldn't parse
• If we can match every register properly, is the work for this repo considered "done" ?
• Are we essentially trying to do the work of a C preprocessor by expanding the DEFINEs to the point where we have a mapping of registers to memory offsets? I might be grossly simplifying it...

Just trying to figure out the end state here.

Also, it seems even if not every register got matched, the SVD file is still usable for the registers that did get matched, right?

[jessebraham]

• The SVD file is a standardized XML file which describes all peripherals for the chip, along with their registers, memory offsets, etc.

Essentially, yes. atsamd-rs has some SVDs from Microchip if you'd like to see some complete examples.

• This project is reading the defines in the esp-idf code, and generating an SVD file
• Currently the program outputs Failed to match reg info at... on registers it couldn't parse

Correct on both. A good chunk of the registers failing to parse seem to be due to inconsistencies in the esp-idf source as far as I can tell.

• If we can match every register properly, is the work for this repo considered "done" ?

As done as anything ever is 😉 esp-idf will continue to be updated I'd imagine so there will likely be tweaks required here and there, but in theory once all peripherals are documented by the SVD file then we have nothing left to do.

• Are we essentially trying to do the work of a C preprocessor by expanding the DEFINEs to the point where we have a mapping of registers to memory offsets? I might be grossly simplifying it...

That seems like a reasonable simplification, though we'd like to get information on interrupts as well (and I'm sure there are further details we can extract).

Also, it seems even if not every register got matched, the SVD file is still usable for the registers that did get matched, right?

The registers which are being matched have PAC code generated for them currently; this is handled by the esp32 repo. In theory if the register is being parsed you should be able to manipulate it programmatically.

Hopefully that helps, anything else I'll leave to @MabezDev

[igrr]

(ESP-IDF maintainer here) We have an issue filed for providing SVD files: espressif/esp-idf#2214. Probably we will solve this for the upcoming ESP32-S2 chip first, because we have started using an internal register description format which can be converted to SVD (some code needs to be written...). For the ESP32, this may take more time because we didn't have that format in place when ESP32 was developed.

I know this doesn't solve the present day issue, just wanted to say that we are aware of this need and sorry for making you jump through hoops to generate SVDs.

[bschwind]

@jessebraham Thanks for the details. Originally I thought maybe using a C preprocessor would be easier, but probably a hand-rolled solution is easy enough so I'll try to work through the code here.

@igrr That's great to hear espressif will have official SVDs at some point!