Tracking issue: Serde and `#[derive(AvroSchema)]` improvements

[Kriskras99]

Our current Serde and derive implementation works, but has room for improvement. This is a unordered list of improvements I'm planning to work on.

• Add documentation for the derive macro (https://github.com/apache/avro-rs/tree/feat/improve_documentation)
  • All supported attributes
  • Compatibility and interaction with Serde
• Make avro(default) (no specific value) use Default::<T>::default() instead of always using None
• Add append_ser support to GenericSingleObjectWriter/to_avro_datum
• Move all Serde related documentation and functions to the serde module (and make that public), so the main documentation is less cluttered (https://github.com/apache/avro-rs/tree/feat/improve_documentation)
  • Also add a short explainer of how Avro works to the Serde part
• Schema aware deserialisation (see feat: Serde driven schema aware deserialization #237)
• Implement SerializeMap::serialize_entry
• Allow disabling getting the rustdoc via doc = false
• In the derived get_schema_with_ctxt validate that the schema is valid (Enums don't serialize correctly #365 (comment))
• Support newtype structs
• Support tuples
• Make clear that errors come from AvroSchema, as that is not visible to the user
• Support char (as a Schema::String or a Schema::Fixed(4))
• Cleanup derive tests to be more like avro_3687_basic_enum_with_default instead of test_avro_3709_record_field_attributes
• Implement AvroSchemaComponent for all types that Serde implements Serialize for
• Allow overwriting the schema type for a field (i.e. avro(type = "bytes"), see apache-avro-derive: how to handle bytes data? #350) (feat(derive): allow overriding the schema for a field and change the schema for uuid::Uuid #397)
• Support #[serde(transparent)] (feat(derive): Support #[serde(transparent)] #398)
• Make AvroSchemaComponent always available (feat!:Make AvroSchemaComponent available without derive feature #394)
• Try to use as many Serde attributes as possible so that user doesn't have duplicate attributes (avro + serde) (feat: Use the Serde attributes and check for conflicting attributes #377)
• Check Serde attributes so we can error on incompatible attributes (i.e. skip_serializing_if without a avro(default) (feat: Use the Serde attributes and check for conflicting attributes #377)

Future work:

• Support simple closures for the with attribute (i.e. #[avro(with = || Schema::Bytes)])

Not possible:

• Use LazyLock to only create the schema once (inspired by feat: Use LazyLock for static schema caching in AvroSchema impl lerouxrgd/rsgen-avro#90)
  • This does not work for types with generics or lifetimes, as the const/static would be shared between all implementations
  • It also doesn't work for types without generics or lifetimes as the output can change based on the names and enclosing_namespace arguments
• Add a test for every derive that runs T::get_schema()
  • This does not work for types with generics or lifetimes, we don't know what bounds are required on the types so we can't name the type.
  • This also doesn't work for types without generics of lifetimes, because derives are only allowed to emit impl blocks, so we can't emit a test function