diff --git a/.gitignore b/.gitignore index 8c6b133a..621540a0 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,5 @@ target /build test-output /dist + +lang/csharp/packages/ diff --git a/CHANGES.txt b/CHANGES.txt index db3e889d..188ec44a 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,6 @@ Avro Change Log -Trunk (not yet released) +Avro 1.7.7 (23 July 2014) NEW FEATURES @@ -20,6 +20,16 @@ Trunk (not yet released) AVRO-1522. Java: Add support for compression codecs to SortedKeyValueFile. (Steven Willis via cutting) + AVRO-1474. C++ resolvind decoder doesn't work when reader schema + has more fields than writer schema. (thiru with help from Ramana + Suvarapu) + + AVRO-1352. Schema for fixed types corrupted when writing out in + JSON format (Steve Roehrs via thiru) + + AVRO-1533. Java: In schema resolution, permit conversion between + bytes and string. (cutting) + OPTIMIZATIONS AVRO-1455. Deep copy does not need to create new instances for primitives. @@ -34,7 +44,7 @@ Trunk (not yet released) (Jesse Anderson via cutting) AVRO-1449. Java: Optionally validate default values while reading schemas. - (cutting) + (cutting) AVRO-1472. Java: Clarify parse method in getting started guide. (Michael Knapp via cutting) @@ -50,6 +60,15 @@ Trunk (not yet released) AVRO-1512. Java: Support Thrift unions. (cutting) + AVRO-1535. Java: Make the name .X to refer to X in the null namespace. + This permits aliases to names in the null namespace. (cutting) + + AVRO-1536. Ruby: Remove monkeypatching of Enumerable. + (Willem van Bergen via martinkl) + + AVRO-1546. Java: Change GenericData.Record#toString() to not + escape forward slashes. (Brandon Forehand via cutting) + BUG FIXES AVRO-1446. C#: Correctly handle system errors in RPC. @@ -91,6 +110,16 @@ Trunk (not yet released) AVRO-1525. Java: ReflectData cannot resolve union with fixed. (tomwhite) + AVRO-1499. Ruby: Fix encoding issue that caused corrupted data files + to be written under Ruby 2.0+. (Willem van Bergen and martinkl) + + AVRO-1498. Java: Fix custom encodings to work in reflect without + Unsafe access. (Christopher Mann via cutting) + + AVRO-1448. Python3: Fix setup.py installation through PyPI. (taton) + + AVRO-1540. C++ doesn't build in Ubuntu. (thiru) + Avro 1.7.6 (15 January 2014) NEW FEATURES @@ -306,7 +335,7 @@ Avro 1.7.5 (12 August 2013) AVRO-1261. Clarify in documentation that generated no-arg constructors do not use default values from schema. (cutting) - AVRO-1297. NettyTransceiver: Provide overloaded + AVRO-1297. NettyTransceiver: Provide overloaded close(boolean awaitCompletion). (jbaldassari) AVRO-1279. C: Treat missing codec in data files as null codec. @@ -468,7 +497,7 @@ Avro 1.7.3 (6 December 2012) NEW FEATURES - AVRO-485. JavaScript: Add validator. (Quinn Slack via cutting) + AVRO-485. JavaScript: Add validator. (Quinn Slack via cutting) AVRO-1157. Java: Extend schema and protocol property support from string-only to full JSON. (cutting) @@ -912,7 +941,7 @@ Avro 1.6.2 (13 February 2012) AVRO-956. Remove dependency on Flex/Bison. (thiru) - AVRO-1011. Improve POM structure. (Lars Francke via scottcarey) + AVRO-1011. Improve POM structure. (Lars Francke via scottcarey) AVRO-1016. Java: Add Field#getAliases() method to better permit copying of schemas. (cutting) @@ -960,7 +989,7 @@ Avro 1.6.2 (13 February 2012) AVRO-966. Java: Fix ReflectDatumWriter to be able to correctly write unions containing Collection and/or ByteBuffer. (cutting) - AVRO-977. Java: Fix codegen to not generate deprecated code. + AVRO-977. Java: Fix codegen to not generate deprecated code. (Hamed Asghari via cutting) AVRO-978. Java: Fix reflect to better handle Byte type. (cutting) @@ -974,10 +1003,10 @@ Avro 1.6.2 (13 February 2012) AVRO-984. C: Resolved writers initialize complex array values correctly. (Vivek Nadkarni via dcreager) - AVRO-994. Java: TestFileSpanStorage.testTonsOfSpans() fails on my + AVRO-994. Java: TestFileSpanStorage.testTonsOfSpans() fails on my slow VM. (jbaldassari) - AVRO-993. Java: Add methods back to GenericDatumReader that were + AVRO-993. Java: Add methods back to GenericDatumReader that were removed in AVRO-839. (jbaldassari) AVRO-1000. Java: Remove incompatible implementations of equals() @@ -1000,7 +1029,7 @@ Avro 1.6.2 (13 February 2012) AVRO-1014. C: Check for errors producing JSON output in avrocat. (Lucas Martin-King via dcreager) - AVRO-996. Java: SpecificRecord builder pattern object copy fails + AVRO-996. Java: SpecificRecord builder pattern object copy fails with unions in some cases. (scottcarey and jbaldassari) AVRO-1020. Java: Fix builder API to correctly handle default @@ -1153,7 +1182,7 @@ Avro 1.6.0 (2 November 2011) AVRO-874. Remove experimental disclaimer from IDL documentation. (cutting) AVRO-891. Java: In SpecificDatumReader, when no reader schema is - specified, use schema of currently loaded class. (cutting) + specified, use schema of currently loaded class. (cutting) AVRO-865. C: Upgrade Jansson to 2.1. (dcreager) @@ -1447,8 +1476,8 @@ Avro 1.5.0 (10 March 2011) AVRO-647. Java: Break avro.jar up into multiple parts: avro.jar, avro-compiler.jar, avro-ipc.jar, avro-mapred.jar, avro-tools.jar, and avro-maven-plugin.jar. - - Summary of artifacts: + + Summary of artifacts: * avro.jar Contains 'core' avro features: schemas, data files, specific, generic, and reflect APIs. @@ -1473,16 +1502,16 @@ Avro 1.5.0 (10 March 2011) (scottcarey) AVRO-737. Java: Improve correlation between packages and modules. - Each module introduced by AVRO-647 now exclusively provides + Each module introduced by AVRO-647 now exclusively provides different java packages. This required moving several classes around into new packages and will therefore require users to change their package imports when upgrading to Avro 1.5.0. Summary of changes: * AvroRemoteException has moved to org.apache.avro - * ByteBufferInputStream and ByteBufferInputStream have moved + * ByteBufferInputStream and ByteBufferInputStream have moved to org.apache.avro.util * InduceSchemaTool has moved to org.apache.avro.tools - * SpecificCompiler, SchemaTask, and ProtocolTask have moved + * SpecificCompiler, SchemaTask, and ProtocolTask have moved to org.apache.avro.compiler.specific * The Idl compiler has moved to org.apache.avro.compiler.idl * ReflectRequestor and ReflectResponder have moved to @@ -1495,20 +1524,20 @@ Avro 1.5.0 (10 March 2011) AVRO-753. Java: Improve BinaryEncoder Performance. The Encoder API has several resulting changes: - * Construction and configuration is handled by EncoderFactory. All + * Construction and configuration is handled by EncoderFactory. All Constructors are hidden, and Encoder.init(OutputStream) is removed. * Some Encoders previously did not buffer output. Users must call Encoder.flush() to ensure output is written unless the EncoderFactory method used to construct an instance explicitly states that the Encoder - does not buffer output. + does not buffer output. (scottcarey) AVRO-769. Java: Align Decoder/Encoder APIs for consistency and long term - stability. Avro's Decoder and Encoder APIs are aligned and now consist of + stability. Avro's Decoder and Encoder APIs are aligned and now consist of only read and write operations. EncoderFactory and DecoderFactory handle all construction and common configuration. Some specialized implementations - have separate configuration APIs. - + have separate configuration APIs. + (scottcarey) AVRO-670. Allow DataFileWriteTool to accept schema files as input with new @@ -1522,7 +1551,7 @@ Avro 1.5.0 (10 March 2011) GenericData, potentially breaking subclasses. (cutting) AVRO-696. Java: Make DataFileWriter.setMetaInternal(String,String) - private. (Patrick Linehan via cutting) + private. (Patrick Linehan via cutting) AVRO-741. C: Minor API change to handling of bytes data. (Douglas Creager via brucem) @@ -1585,7 +1614,7 @@ Avro 1.5.0 (10 March 2011) AVRO-765. Java: Improvement to BinaryDecoder readLong performance (scottcarey) - + AVRO-716. Java: integrate AVRO-647 changes with top level build (scottcarey) @@ -1663,7 +1692,7 @@ Avro 1.5.0 (10 March 2011) AVRO-764. Java: Bug in BinaryData.compare() with offset comparison. (Harsh J Chouraria via scottcarey) - AVRO-743. Java: Performance Regression and memory pressure with + AVRO-743. Java: Performance Regression and memory pressure with GenericDatumReader. (scottcarey) AVRO-675. C: Bytes and fixed setters don't update datum size. @@ -1693,7 +1722,7 @@ Avro 1.5.0 (10 March 2011) (Bo Shi via cutting) AVRO-713. Java: Fix GenericData.Record#toString() to produce valid - JSON for enum symbols. (Jay Kreps via cutting) + JSON for enum symbols. (Jay Kreps via cutting) AVRO-643. Java: Fix intermittent failures in TestTraceCollection. (cutting) @@ -1841,7 +1870,7 @@ Avro 1.4.0 (31 August 2010) IMPROVEMENTS - AVRO-636. Expose Singleton Method for TracePlugin. (Patrick Wendell via + AVRO-636. Expose Singleton Method for TracePlugin. (Patrick Wendell via philz) AVRO-614. Improve Trace frontend UI. (Patrick Wendell via philz) @@ -1855,7 +1884,7 @@ Avro 1.4.0 (31 August 2010) AVRO-584. Update Histogram for Stats Plugin (Patrick Wendell via philz) - AVRO-501. missing function in C api to access array elements after + AVRO-501. missing function in C api to access array elements after decoding an array. (Bruce Mitchener via massie) AVRO-497. Minor changes to C++ autotools, makefiles, and code @@ -1941,7 +1970,7 @@ Avro 1.4.0 (31 August 2010) AVRO-622. python avro.ipc doesn't work with python2.4 (philz) - AVRO-620. Python implementation doesn't stringify sub-schemas + AVRO-620. Python implementation doesn't stringify sub-schemas correctly. (philz) AVRO-618. Avro doesn't work with python 2.4 (philz) @@ -1956,7 +1985,7 @@ Avro 1.4.0 (31 August 2010) AVRO-566. Java: fix so that JAVA_HOME is bound by build.xml for test_tools.sh. (cutting) - AVRO-571. Fix how we handle out-of-bounds indexes for union and + AVRO-571. Fix how we handle out-of-bounds indexes for union and enum parsing in Python (hammer) AVRO-589. ClassCastException: @@ -2008,7 +2037,7 @@ Avro 1.3.3 (7 June 2010) AVRO-486. DataFile.open for the ruby side (jmhodges) - AVRO-559. Handle read_union error where the list index of the union branch + AVRO-559. Handle read_union error where the list index of the union branch to follow exceeds the size of the union schema (hammer) AVRO-491. Doing doubles and floats better in the ruby impl. (jmhodges) @@ -2045,7 +2074,7 @@ Avro 1.3.3 (7 June 2010) AVRO-562 ruby side had busted client handshaking. (jmhodges) - AVRO-517. Resolving Decoder fails in some cases. (thiru) + AVRO-517. Resolving Decoder fails in some cases. (thiru) AVRO-524. DataFileWriter.appendTo leads to intermittent IOException during write() (thiru) @@ -2060,14 +2089,14 @@ Avro 1.3.3 (7 June 2010) AVRO-500. ruby side dev packaging (jmhodges) AVRO-516. ruby: buffer length should not be little-endian in socket rpc (jmhodges) - + Avro 1.3.2 (31 March 2010) IMPROVEMENTS AVRO-449. CMake-based build system for Avro/C (Bruce Mitchener via massie) - AVRO-418. avro.h generates errors when included in C++ code + AVRO-418. avro.h generates errors when included in C++ code (Bruce Mitchener via massie) AVRO-480. avro_flush() is in the header, but not implemented @@ -2084,7 +2113,7 @@ Avro 1.3.2 (31 March 2010) AVRO-490. Add Ant task to deploy Java artifacts to Maven repo. (cutting) BUG FIXES - + AVRO-479. Fix 'sign' target in top-level build.sh to generate md5 checksums. (cutting) @@ -2106,15 +2135,15 @@ Avro 1.3.1 (16 March 2010) AVRO-438. Clarify spec. (Amichai Rothman via cutting) - AVRO-445. avro_size_data() to pre-calculate the size of an + AVRO-445. avro_size_data() to pre-calculate the size of an avro_datum_t in serialized form (Bruce Mitchener via massie) - AVRO-443. Endianness is determined at configure time rather + AVRO-443. Endianness is determined at configure time rather than compile time (Bruce Mitchener via massie) - + AVRO-448. encoding_binary.c doesn't build on big endian platforms (Bruce Mitchener via massie) - + AVRO-442. sizeof void* and sizeof long detected at configure time (Bruce Mitchener via massie) @@ -2204,7 +2233,7 @@ Avro 1.3.0 (24 February 2010) - jsontofrag Renders a JSON-encoded Avro datum as binary. (Philip Zeyliger via cutting) - AVRO-272. Extend RPCContext to include message. + AVRO-272. Extend RPCContext to include message. (Philip Zeyliger via cutting) AVRO-258. Add GenAvro language tool. (Todd Lipcon via cutting) @@ -2221,20 +2250,20 @@ Avro 1.3.0 (24 February 2010) protocols, and messages. (Philip Zeyliger via cutting) AVRO-274. Make Java's data file sync interval configurable. (cutting) - + AVRO-346. Add function to validate a datum against a schema. (massie) AVRO-306. Add Ruby implementation. (Jeff Hodges via cutting) - + AVRO-135. Add compression to data files. (philz) - AVRO-368. Reserve avro.* in object container files, and + AVRO-368. Reserve avro.* in object container files, and rename existing reserved words. (philz) AVRO-380. Avro Container File format change: add block size to block descriptor. (Scott Carey via philz) - AVRO-322. Add a working client and server to Python implementation + AVRO-322. Add a working client and server to Python implementation using HTTP as a transport (hammer) AVRO-287. Make RPC interop tests work with new Python implementation @@ -2273,7 +2302,7 @@ Avro 1.3.0 (24 February 2010) AVRO-149. Add Java command-line executable, "avroj". (Philip Zeyliger via cutting) - AVRO-175. Split the avro_io interface into two interfaces: avro_reader + AVRO-175. Split the avro_io interface into two interfaces: avro_reader and avro_writer (massie) AVRO-179. Add units tests for all Avro C primitives (massie) @@ -2284,7 +2313,7 @@ Avro 1.3.0 (24 February 2010) AVRO-186. Full read-path interoperability test (massie) - AVRO-187. Move top-level source files into separate directories + AVRO-187. Move top-level source files into separate directories for easier maintenance (massie) AVRO-188. Need to update svn ignores (massie) @@ -2327,7 +2356,7 @@ Avro 1.3.0 (24 February 2010) AVRO-236. Add protocol support to avroj induce tool. (cutting) AVRO-234. C++ code cleanup. (sbanacho) - + AVRO-240. In Python, if simplejson is not available, try using 2.6's built-in json module. (Jeff Hammerbacher via cutting) @@ -2341,7 +2370,7 @@ Avro 1.3.0 (24 February 2010) AVRO-247. In reflection, add Stringable annotation to indicate classes that can be represented by an Avro string. (cutting) - + AVRO-246 Java schema parser should take schema from InputStream in addition to file. (thiru) @@ -2426,12 +2455,12 @@ Avro 1.3.0 (24 February 2010) AVRO-377. Add getters and setters for all Avro datum types (massie) - AVRO-378. Add example code to the C implementation and update + AVRO-378. Add example code to the C implementation and update documentation (massie) AVRO-379. Changed record getter/setter API to match other datatypes (massie) - AVRO-381. Update documentation to talk about reference counting and + AVRO-381. Update documentation to talk about reference counting and memory management (massie) AVRO-384. Add schema projection to the C implementation (massie) @@ -2460,7 +2489,7 @@ Avro 1.3.0 (24 February 2010) AVRO-321. Restore Java RPC interop tests. (cutting) - AVRO-402. Add method for writing avro_schema_t structure to an + AVRO-402. Add method for writing avro_schema_t structure to an avro_writer_t (massie) AVRO-398. avro_read_file doesn't detect eof (Eli Collins via massie) @@ -2469,7 +2498,7 @@ Avro 1.3.0 (24 February 2010) AVRO-400. Adding warning for unused parameters (Eli Collins via massie) - AVRO-409. Update contact database example to use a file object + AVRO-409. Update contact database example to use a file object container for C implementation (massie) AVRO-420. Add namespace support to C implementation (massie) @@ -2503,7 +2532,7 @@ Avro 1.3.0 (24 February 2010) AVRO-345. Optimization for ResolvingDecoder (thiru) - AVRO-363. estSchema had two tests disabled; new test for named schemas + AVRO-363. estSchema had two tests disabled; new test for named schemas named after primitives. (philz) AVRO-354. Performance improvement to BinaryDecoder.readInt() (Kevin Oliver via thiru) @@ -2521,7 +2550,7 @@ Avro 1.3.0 (24 February 2010) performance. (Scott Carey via cutting) BUG FIXES - + AVRO-176. Safeguard against bad istreams before reading. (sbanacho) AVRO-141. Fix a NullPointerException in ReflectData#isRecord(). @@ -2550,7 +2579,7 @@ Avro 1.3.0 (24 February 2010) AVRO-194. C++ varint encoding buffer too small. (sbanacho) - AVRO-210. Memory leak with recursive schemas when constructed + AVRO-210. Memory leak with recursive schemas when constructed by hand. (sbanacho) AVRO-211. Nested schema does not get parsed in C++. (sbanacho) @@ -2560,7 +2589,7 @@ Avro 1.3.0 (24 February 2010) AVRO-223. Fix test-avroj on Mac OS X. (Philip Zeyliger via cutting) - AVRO-224. Code cleanup: cleaner distinction between public and private + AVRO-224. Code cleanup: cleaner distinction between public and private methods (massie) AVRO-221. Mangle Java reserved words in generated code to avoid @@ -2590,7 +2619,7 @@ Avro 1.3.0 (24 February 2010) AVRO-280. Fix file header schema in specification. Also fix "forrestdoc" build target to work on clean checkout. - (Jeff Hammerbacher & cutting) + (Jeff Hammerbacher & cutting) AVRO-292. Fix Python skipping of ints and longs. (Jeff Hammerbacher via cutting) @@ -2610,7 +2639,7 @@ Avro 1.3.0 (24 February 2010) AVRO-47. Use void* for byte sequences. (sbanacho) AVRO-337. ant test-java fails in Cygwin due to CRLF v LF problem (thiru) - + AVRO-347. Add the --unsafe flag to asciidoc in order to include source/header files (massie) AVRO-352. Incorrect binary encoding for strings and bytes (massie) @@ -2626,7 +2655,7 @@ Avro 1.3.0 (24 February 2010) AVRO-382. Avro hashCode throws a NullPointerException when fields are uninitialized (Michael Armbrust via philz) AVRO-385. Initializing uninizialized BlockingBinaryEncoder fails (thiru) - + AVRO-389. ResolvingDecoder does not resolve enum well (thiru) AVRO-390. ResolvingDecoder does not handle default values for records well (thiru) @@ -2920,7 +2949,7 @@ Avro 1.0.0 -- 9 July 2009 AVRO-13. Use dictionary instead of if-else in validate. (sharad) - AVRO-5. Add java versus python RPC interoperability tests. + AVRO-5. Add java versus python RPC interoperability tests. (sharad) AVRO-16. Minor documentation improvements. (cutting) @@ -2954,7 +2983,7 @@ Avro 1.0.0 -- 9 July 2009 AVRO-68. Add license headers to C sources and improve C packaging. (Matt Massie via cutting) - AVRO-351. Shorten induce tool description; add check to avoid overly verbose + AVRO-351. Shorten induce tool description; add check to avoid overly verbose descriptions. (philz) OPTIMIZATIONS @@ -2963,7 +2992,7 @@ Avro 1.0.0 -- 9 July 2009 AVRO-3. Fix ValueReader to throw an exception at EOF. (Pat Hunt via cutting) - + AVRO-4. Fix so that specific code generation works under Eclipse. (Pat Hunt via cutting) @@ -2992,7 +3021,7 @@ Avro 1.0.0 -- 9 July 2009 AVRO-45. Fix c++ compliation so that python script need not be made executable. (Scott Banachowski via cutting) - AVRO-51. Fix testio.py to exit correctly. (Philip Zeyliger + AVRO-51. Fix testio.py to exit correctly. (Philip Zeyliger via sharad) AVRO-55. Fix two spec document typos. (cutting) diff --git a/build.sh b/build.sh index 25a62d13..06961c0e 100755 --- a/build.sh +++ b/build.sh @@ -117,7 +117,7 @@ case "$target" in (cd lang/php; ./build.sh dist) mkdir -p dist/perl - (cd lang/perl; make dist) + (cd lang/perl; perl ./Makefile.PL && make dist) cp lang/perl/Avro-$VERSION.tar.gz dist/perl/ # build docs @@ -169,7 +169,7 @@ case "$target" in (cd lang/php; ./build.sh clean) - (cd lang/perl; make clean) + (cd lang/perl; [ -f Makefile ] && make clean) ;; *) diff --git a/doc/src/content/xdocs/spec.xml b/doc/src/content/xdocs/spec.xml index ec5280e6..8c108c88 100644 --- a/doc/src/content/xdocs/spec.xml +++ b/doc/src/content/xdocs/spec.xml @@ -259,6 +259,8 @@
  • subsequently contain only [A-Za-z0-9_]

    • A namespace is a dot-separated sequence of such names. + The empty string may also be used as a namespace to indicate the + null namespace. Equality of names (including field names and enum symbols) as well as fullnames is case-sensitive.

    In record, enum and fixed definitions, the fullname is @@ -279,7 +281,8 @@ if "name": "X" is specified, and this occurs within a field of the record definition of org.foo.Y, then the fullname - is org.foo.X. + is org.foo.X. If there is no enclosing + namespace then the null namespace is used.

    References to previously defined names are as in the latter two cases above: if they contain a dot they are a fullname, if @@ -1076,6 +1079,8 @@

  • int is promotable to long, float, or double
  • long is promotable to float or double
  • float is promotable to double
    • +
  • string is promotable to bytes
    • +
  • bytes is promotable to string
    • diff --git a/lang/c++/.gitignore b/lang/c++/.gitignore index 9a0682e5..091cd7f5 100644 --- a/lang/c++/.gitignore +++ b/lang/c++/.gitignore @@ -1,6 +1,2 @@ -.svn/ build/ -build.win/ -doc/ -m4/ -test.avro +build.mac/ diff --git a/lang/c++/CMakeLists.txt b/lang/c++/CMakeLists.txt index 81609430..fc54a71e 100644 --- a/lang/c++/CMakeLists.txt +++ b/lang/c++/CMakeLists.txt @@ -42,6 +42,7 @@ add_definitions ( -DBOOST_REGEX_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_SYSTEM_DYN_LINK + -DBOOST_IOSTREAMS_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_ALL_NO_LIB) endif() @@ -64,7 +65,7 @@ set (AVRO_SOURCE_FILES impl/Types.cc impl/ValidSchema.cc impl/Zigzag.cc impl/BinaryEncoder.cc impl/BinaryDecoder.cc impl/Stream.cc impl/FileStream.cc - impl/Generic.cc + impl/Generic.cc impl/GenericDatum.cc impl/DataFile.cc impl/parsing/Symbol.cc impl/parsing/ValidatingCodec.cc @@ -97,21 +98,6 @@ add_executable (precompile test/precompile.cc) target_link_libraries (precompile avrocpp_s ${Boost_LIBRARIES}) -macro (gencpp file ns) - add_custom_command (OUTPUT ${ns}.hh - COMMAND precompile ${CMAKE_CURRENT_SOURCE_DIR}/jsonschemas/${file} - ${file} - COMMAND python ${CMAKE_CURRENT_SOURCE_DIR}/scripts/gen-cppcode.py - -n ${ns} -i ${file} -o ${ns}.hh - DEPENDS precompile ${CMAKE_CURRENT_SOURCE_DIR}/jsonschemas/${file}) - add_custom_target(${ns} DEPENDS ${ns}.hh) -endmacro (gencpp) - -if (CYGWIN OR NOT WIN32) - gencpp (bigrecord testgen) - gencpp (bigrecord2 testgen2) -endif () - macro (gen file ns) add_custom_command (OUTPUT ${file}.hh COMMAND avrogencpp @@ -124,6 +110,7 @@ endmacro (gen) gen (empty_record empty) gen (bigrecord testgen) +gen (bigrecord_r testgen_r) gen (bigrecord2 testgen2) gen (tweet testgen3) gen (union_array_union uau) @@ -156,12 +143,8 @@ unittest (DataFileTests) unittest (JsonTests) unittest (AvrogencppTests) -if (CYGWIN OR NOT WIN32) - unittest (testgentest) - add_dependencies (testgentest testgen testgen2) -endif() - -add_dependencies (AvrogencppTests bigrecord_hh bigrecord2_hh tweet_hh +add_dependencies (AvrogencppTests bigrecord_hh bigrecord_r_hh bigrecord2_hh + tweet_hh union_array_union_hh union_map_union_hh union_conflict_hh recursive_hh reuse_hh circulardep_hh empty_record_hh) diff --git a/lang/c++/api/Generic.hh b/lang/c++/api/Generic.hh index 4edbb8a6..ae6cb09e 100644 --- a/lang/c++/api/Generic.hh +++ b/lang/c++/api/Generic.hh @@ -19,470 +19,15 @@ #ifndef avro_Generic_hh__ #define avro_Generic_hh__ -#include -#include -#include - -#include #include #include "Config.hh" -#include "Node.hh" #include "Types.hh" #include "Encoder.hh" #include "Decoder.hh" -#include "ValidSchema.hh" +#include "GenericDatum.hh" namespace avro { -/** - * Generic datum which can hold any Avro type. The datum has a type - * and a value. The type is one of the Avro data types. The C++ type for - * value corresponds to the Avro type. - * \li An Avro null corresponds to no C++ type. It is illegal to - * to try to access values for null. - * \li Avro boolean maps to C++ bool - * \li Avro int maps to C++ int32_t. - * \li Avro long maps to C++ int64_t. - * \li Avro float maps to C++ float. - * \li Avro double maps to C++ double. - * \li Avro string maps to C++ std::string. - * \li Avro bytes maps to C++ std::vector<uint_t>. - * \li Avro fixed maps to C++ class GenericFixed. - * \li Avro enum maps to C++ class GenericEnum. - * \li Avro array maps to C++ class GenericArray. - * \li Avro map maps to C++ class GenericMap. - * \li There is no C++ type corresponding to Avro union. The - * object should have the C++ type corresponing to one of the constituent - * types of the union. - * - */ -class AVRO_DECL GenericDatum { - Type type_; - boost::any value_; - - GenericDatum(Type t) : type_(t) { } - - template - GenericDatum(Type t, const T& v) : type_(t), value_(v) { } - - void init(const NodePtr& schema); -public: - /** - * The avro data type this datum holds. - */ - Type type() const; - - /** - * Returns the value held by this datum. - * T The type for the value. This must correspond to the - * avro type returned by type(). - */ - template const T& value() const; - - /** - * Returns the reference to the value held by this datum, which - * can be used to change the contents. Please note that only - * value can be changed, the data type of the value held cannot - * be changed. - * - * T The type for the value. This must correspond to the - * avro type returned by type(). - */ - template T& value(); - - /** - * Returns true if and only if this datum is a union. - */ - bool isUnion() const { return type_ == AVRO_UNION; } - - /** - * Returns the index of the current branch, if this is a union. - * \sa isUnion(). - */ - size_t unionBranch() const; - - /** - * Selects a new branch in the union if this is a union. - * \sa isUnion(). - */ - void selectBranch(size_t branch); - - /// Makes a new AVRO_NULL datum. - GenericDatum() : type_(AVRO_NULL) { } - - /// Makes a new AVRO_BOOL datum whose value is of type bool. - GenericDatum(bool v) : type_(AVRO_BOOL), value_(v) { } - - /// Makes a new AVRO_INT datum whose value is of type int32_t. - GenericDatum(int32_t v) : type_(AVRO_INT), value_(v) { } - - /// Makes a new AVRO_LONG datum whose value is of type int64_t. - GenericDatum(int64_t v) : type_(AVRO_LONG), value_(v) { } - - /// Makes a new AVRO_FLOAT datum whose value is of type float. - GenericDatum(float v) : type_(AVRO_FLOAT), value_(v) { } - - /// Makes a new AVRO_DOUBLE datum whose value is of type double. - GenericDatum(double v) : type_(AVRO_DOUBLE), value_(v) { } - - /// Makes a new AVRO_STRING datum whose value is of type std::string. - GenericDatum(const std::string& v) : type_(AVRO_STRING), value_(v) { } - - /// Makes a new AVRO_BYTES datum whose value is of type - /// std::vector. - GenericDatum(const std::vector& v) : - type_(AVRO_BYTES), value_(v) { } - - /** - * Constructs a datum corresponding to the given avro type. - * The value will the appropraite default corresponding to the - * data type. - * \param schema The schema that defines the avro type. - */ - GenericDatum(const NodePtr& schema); - - /** - * Constructs a datum corresponding to the given avro type. - * The value will the appropraite default corresponding to the - * data type. - * \param schema The schema that defines the avro type. - */ - GenericDatum(const ValidSchema& schema); -}; - -/** - * The base class for all generic type for containers. - */ -class AVRO_DECL GenericContainer { - NodePtr schema_; - static void assertType(const NodePtr& schema, Type type); -protected: - /** - * Constructs a container corresponding to the given schema. - */ - GenericContainer(Type type, const NodePtr& s) : schema_(s) { - assertType(s, type); - } - -public: - /// Returns the schema for this object - const NodePtr& schema() const { - return schema_; - } -}; - -/** - * Generic container for unions. - */ -class AVRO_DECL GenericUnion : public GenericContainer { - size_t curBranch_; - GenericDatum datum_; - -public: - /** - * Constructs a generic union corresponding to the given schema \p schema, - * and the given value. The schema should be of Avro type union - * and the value should correspond to one of the branches of the union. - */ - GenericUnion(const NodePtr& schema) : - GenericContainer(AVRO_UNION, schema), curBranch_(schema->leaves()) { - } - - /** - * Returns the index of the current branch. - */ - size_t currentBranch() const { return curBranch_; } - - /** - * Selects a new branch. The type for the value is changed accordingly. - * \param branch The index for the selected branch. - */ - void selectBranch(size_t branch) { - if (curBranch_ != branch) { - datum_ = GenericDatum(schema()->leafAt(branch)); - curBranch_ = branch; - } - } - - /** - * Returns the type for currently selected branch in this union. - */ - Type type() const { - return datum_.type(); - } - - /** - * Returns the value in this union. - */ - template - const T& value() const { - return datum_.value(); - } - - /** - * Returns the reference to the value in this union. - */ - template - T& value() { - return datum_.value(); - } - -}; - -/** - * The generic container for Avro records. - */ -class AVRO_DECL GenericRecord : public GenericContainer { - std::vector fields_; -public: - /** - * Constructs a generic record corresponding to the given schema \p schema, - * which should be of Avro type record. - */ - GenericRecord(const NodePtr& schema); - - /** - * Returns the number of fields in the current record. - */ - size_t fieldCount() const { - return fields_.size(); - } - - /** - * Returns index of the field with the given name \p name - */ - size_t fieldIndex(const std::string& name) const { - size_t index = 0; - if (!schema()->nameIndex(name, index)) { - throw Exception("Invalid field name: " + name); - } - return index; - } - - /** - * Returns true if a field with the given name \p name is located in this record, - * false otherwise - */ - bool hasField(const std::string& name) const { - size_t index = 0; - return schema()->nameIndex(name, index); - } - - /** - * Returns the field with the given name \p name. - */ - const GenericDatum& field(const std::string& name) const { - return fieldAt(fieldIndex(name)); - } - - /** - * Returns the reference to the field with the given name \p name, - * which can be used to change the contents. - */ - GenericDatum& field(const std::string& name) { - return fieldAt(fieldIndex(name)); - } - - /** - * Returns the field at the given position \p pos. - */ - const GenericDatum& fieldAt(size_t pos) const { - return fields_[pos]; - } - - /** - * Returns the reference to the field at the given position \p pos, - * which can be used to change the contents. - */ - GenericDatum& fieldAt(size_t pos) { - return fields_[pos]; - } - - /** - * Replaces the field with the given name \p name with \p v. - */ - void setField(const std::string& name, const GenericDatum& v) { - // assertSameType(v, schema()->leafAt(pos)); - return setFieldAt(fieldIndex(name), v); - } - - /** - * Replaces the field at the given position \p pos with \p v. - */ - void setFieldAt(size_t pos, const GenericDatum& v) { - // assertSameType(v, schema()->leafAt(pos)); - fields_[pos] = v; - } -}; - -/** - * The generic container for Avro arrays. - */ -class AVRO_DECL GenericArray : public GenericContainer { -public: - /** - * The contents type for the array. - */ - typedef std::vector Value; - - /** - * Constructs a generic array corresponding to the given schema \p schema, - * which should be of Avro type array. - */ - GenericArray(const NodePtr& schema) : GenericContainer(AVRO_ARRAY, schema) { - } - - /** - * Returns the contents of this array. - */ - const Value& value() const { - return value_; - } - - /** - * Returns the reference to the contents of this array. - */ - Value& value() { - return value_; - } -private: - Value value_; -}; - -/** - * The generic container for Avro maps. - */ -class AVRO_DECL GenericMap : public GenericContainer { -public: - /** - * The contents type for the map. - */ - typedef std::vector > Value; - - /** - * Constructs a generic map corresponding to the given schema \p schema, - * which should be of Avro type map. - */ - GenericMap(const NodePtr& schema) : GenericContainer(AVRO_MAP, schema) { - } - - /** - * Returns the contents of this map. - */ - const Value& value() const { - return value_; - } - - /** - * Returns the reference to the contents of this map. - */ - Value& value() { - return value_; - } -private: - Value value_; -}; - -/** - * Generic container for Avro enum. - */ -class AVRO_DECL GenericEnum : public GenericContainer { - size_t value_; -public: - /** - * Constructs a generic enum corresponding to the given schema \p schema, - * which should be of Avro type enum. - */ - GenericEnum(const NodePtr& schema) : - GenericContainer(AVRO_ENUM, schema), value_(0) { - } - - /** - * Returns the symbol corresponding to the cardinal \p n. If the - * value for \p n is not within the limits an exception is thrown. - */ - const std::string& symbol(size_t n) { - if (n < schema()->names()) { - return schema()->nameAt(n); - } - throw Exception("Not as many symbols"); - } - - /** - * Returns the cardinal for the given symbol \c symbol. If the symbol - * is not defined for this enum and exception is thrown. - */ - size_t index(const std::string& symbol) const { - size_t result; - if (schema()->nameIndex(symbol, result)) { - return result; - } - throw Exception("No such symbol"); - } - - /** - * Set the value for this enum corresponding to the given symbol \c symbol. - */ - size_t set(const std::string& symbol) { - return value_ = index(symbol); - } - - /** - * Set the value for this enum corresponding to the given cardinal \c n. - */ - void set(size_t n) { - if (n < schema()->names()) { - value_ = n; - return; - } - throw Exception("Not as many symbols"); - } - - /** - * Returns the cardinal for the current value of this enum. - */ - size_t value() const { - return value_; - } - - /** - * Returns the symbol for the current value of this enum. - */ - const std::string& symbol() const { - return schema()->nameAt(value_); - } -}; - -/** - * Generic container for Avro fixed. - */ -class AVRO_DECL GenericFixed : public GenericContainer { - std::vector value_; -public: - /** - * Constructs a generic enum corresponding to the given schema \p schema, - * which should be of Avro type fixed. - */ - GenericFixed(const NodePtr& schema) : GenericContainer(AVRO_FIXED, schema) { - value_.resize(schema->fixedSize()); - } - - /** - * Returns the contents of this fixed. - */ - const std::vector& value() const { - return value_; - } - - /** - * Returns the reference to the contents of this fixed. - */ - std::vector& value() { - return value_; - } -}; - - /** * A utility class to read generic datum from decoders. */ @@ -556,33 +101,6 @@ public: } }; -inline Type AVRO_DECL GenericDatum::type() const { - return (type_ == AVRO_UNION) ? - boost::any_cast(&value_)->type() : type_; -} - -template -const T& GenericDatum::value() const { - return (type_ == AVRO_UNION) ? - boost::any_cast(&value_)->value() : - *boost::any_cast(&value_); -} - -template -T& GenericDatum::value() { - return (type_ == AVRO_UNION) ? - boost::any_cast(&value_)->value() : - *boost::any_cast(&value_); -} - -inline size_t GenericDatum::unionBranch() const { - return boost::any_cast(&value_)->currentBranch(); -} - -inline void GenericDatum::selectBranch(size_t branch) { - boost::any_cast(&value_)->selectBranch(branch); -} - template struct codec_traits; /** diff --git a/lang/c++/api/GenericDatum.hh b/lang/c++/api/GenericDatum.hh new file mode 100644 index 00000000..5efcb7f4 --- /dev/null +++ b/lang/c++/api/GenericDatum.hh @@ -0,0 +1,505 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef avro_GenericDatum_hh__ +#define avro_GenericDatum_hh__ + +#include +#include +#include +#include + +#include + +#include "Node.hh" +#include "ValidSchema.hh" + +namespace avro { +/** + * Generic datum which can hold any Avro type. The datum has a type + * and a value. The type is one of the Avro data types. The C++ type for + * value corresponds to the Avro type. + * \li An Avro null corresponds to no C++ type. It is illegal to + * to try to access values for null. + * \li Avro boolean maps to C++ bool + * \li Avro int maps to C++ int32_t. + * \li Avro long maps to C++ int64_t. + * \li Avro float maps to C++ float. + * \li Avro double maps to C++ double. + * \li Avro string maps to C++ std::string. + * \li Avro bytes maps to C++ std::vector<uint_t>. + * \li Avro fixed maps to C++ class GenericFixed. + * \li Avro enum maps to C++ class GenericEnum. + * \li Avro array maps to C++ class GenericArray. + * \li Avro map maps to C++ class GenericMap. + * \li There is no C++ type corresponding to Avro union. The + * object should have the C++ type corresponing to one of the constituent + * types of the union. + * + */ +class AVRO_DECL GenericDatum { + Type type_; + boost::any value_; + + GenericDatum(Type t) : type_(t) { } + + template + GenericDatum(Type t, const T& v) : type_(t), value_(v) { } + + void init(const NodePtr& schema); +public: + /** + * The avro data type this datum holds. + */ + Type type() const { + return type_; + } + + /** + * Returns the value held by this datum. + * T The type for the value. This must correspond to the + * avro type returned by type(). + */ + template const T& value() const { + return *boost::any_cast(&value_); + } + + /** + * Returns the reference to the value held by this datum, which + * can be used to change the contents. Please note that only + * value can be changed, the data type of the value held cannot + * be changed. + * + * T The type for the value. This must correspond to the + * avro type returned by type(). + */ + template T& value() { + return *boost::any_cast(&value_); + } + + /** + * Returns true if and only if this datum is a union. + */ + bool isUnion() const { return type_ == AVRO_UNION; } + + /** + * Returns the index of the current branch, if this is a union. + * \sa isUnion(). + */ + size_t unionBranch() const; + + /** + * Selects a new branch in the union if this is a union. + * \sa isUnion(). + */ + void selectBranch(size_t branch); + + /// Makes a new AVRO_NULL datum. + GenericDatum() : type_(AVRO_NULL) { } + + /// Makes a new AVRO_BOOL datum whose value is of type bool. + GenericDatum(bool v) : type_(AVRO_BOOL), value_(v) { } + + /// Makes a new AVRO_INT datum whose value is of type int32_t. + GenericDatum(int32_t v) : type_(AVRO_INT), value_(v) { } + + /// Makes a new AVRO_LONG datum whose value is of type int64_t. + GenericDatum(int64_t v) : type_(AVRO_LONG), value_(v) { } + + /// Makes a new AVRO_FLOAT datum whose value is of type float. + GenericDatum(float v) : type_(AVRO_FLOAT), value_(v) { } + + /// Makes a new AVRO_DOUBLE datum whose value is of type double. + GenericDatum(double v) : type_(AVRO_DOUBLE), value_(v) { } + + /// Makes a new AVRO_STRING datum whose value is of type std::string. + GenericDatum(const std::string& v) : type_(AVRO_STRING), value_(v) { } + + /// Makes a new AVRO_BYTES datum whose value is of type + /// std::vector. + GenericDatum(const std::vector& v) : + type_(AVRO_BYTES), value_(v) { } + + /** + * Constructs a datum corresponding to the given avro type. + * The value will the appropraite default corresponding to the + * data type. + * \param schema The schema that defines the avro type. + */ + GenericDatum(const NodePtr& schema); + + /** + * Constructs a datum corresponding to the given avro type and set + * the value. + * \param schema The schema that defines the avro type. + * \param v The value for this type. + */ + template + GenericDatum(const NodePtr& schema, const T& v) : + type_(schema->type()) { + init(schema); + value() = v; + } + + /** + * Constructs a datum corresponding to the given avro type. + * The value will the appropraite default corresponding to the + * data type. + * \param schema The schema that defines the avro type. + */ + GenericDatum(const ValidSchema& schema); +}; + +/** + * The base class for all generic type for containers. + */ +class AVRO_DECL GenericContainer { + NodePtr schema_; + static void assertType(const NodePtr& schema, Type type); +protected: + /** + * Constructs a container corresponding to the given schema. + */ + GenericContainer(Type type, const NodePtr& s) : schema_(s) { + assertType(s, type); + } + +public: + /// Returns the schema for this object + const NodePtr& schema() const { + return schema_; + } +}; + +/** + * Generic container for unions. + */ +class AVRO_DECL GenericUnion : public GenericContainer { + size_t curBranch_; + GenericDatum datum_; + +public: + /** + * Constructs a generic union corresponding to the given schema \p schema, + * and the given value. The schema should be of Avro type union + * and the value should correspond to one of the branches of the union. + */ + GenericUnion(const NodePtr& schema) : + GenericContainer(AVRO_UNION, schema), curBranch_(schema->leaves()) { + } + + /** + * Returns the index of the current branch. + */ + size_t currentBranch() const { return curBranch_; } + + /** + * Selects a new branch. The type for the value is changed accordingly. + * \param branch The index for the selected branch. + */ + void selectBranch(size_t branch) { + if (curBranch_ != branch) { + datum_ = GenericDatum(schema()->leafAt(branch)); + curBranch_ = branch; + } + } + + /** + * Returns the datum corresponding to the currently selected branch + * in this union. + */ + GenericDatum& datum() { + return datum_; + } + + /** + * Returns the datum corresponding to the currently selected branch + * in this union. + */ + const GenericDatum& datum() const { + return datum_; + } +}; + +/** + * The generic container for Avro records. + */ +class AVRO_DECL GenericRecord : public GenericContainer { + std::vector fields_; +public: + /** + * Constructs a generic record corresponding to the given schema \p schema, + * which should be of Avro type record. + */ + GenericRecord(const NodePtr& schema); + + /** + * Returns the number of fields in the current record. + */ + size_t fieldCount() const { + return fields_.size(); + } + + /** + * Returns index of the field with the given name \p name + */ + size_t fieldIndex(const std::string& name) const { + size_t index = 0; + if (!schema()->nameIndex(name, index)) { + throw Exception("Invalid field name: " + name); + } + return index; + } + + /** + * Returns true if a field with the given name \p name is located in this r + * false otherwise + */ + bool hasField(const std::string& name) const { + size_t index = 0; + return schema()->nameIndex(name, index); + } + + /** + * Returns the field with the given name \p name. + */ + const GenericDatum& field(const std::string& name) const { + return fieldAt(fieldIndex(name)); + } + + /** + * Returns the reference to the field with the given name \p name, + * which can be used to change the contents. + */ + GenericDatum& field(const std::string& name) { + return fieldAt(fieldIndex(name)); + } + + /** + * Returns the field at the given position \p pos. + */ + const GenericDatum& fieldAt(size_t pos) const { + return fields_[pos]; + } + + /** + * Returns the reference to the field at the given position \p pos, + * which can be used to change the contents. + */ + GenericDatum& fieldAt(size_t pos) { + return fields_[pos]; + } + + /** + * Replaces the field at the given position \p pos with \p v. + */ + void setFieldAt(size_t pos, const GenericDatum& v) { + // assertSameType(v, schema()->leafAt(pos)); + fields_[pos] = v; + } +}; + +/** + * The generic container for Avro arrays. + */ +class AVRO_DECL GenericArray : public GenericContainer { +public: + /** + * The contents type for the array. + */ + typedef std::vector Value; + + /** + * Constructs a generic array corresponding to the given schema \p schema, + * which should be of Avro type array. + */ + GenericArray(const NodePtr& schema) : GenericContainer(AVRO_ARRAY, schema) { + } + + /** + * Returns the contents of this array. + */ + const Value& value() const { + return value_; + } + + /** + * Returns the reference to the contents of this array. + */ + Value& value() { + return value_; + } +private: + Value value_; +}; + +/** + * The generic container for Avro maps. + */ +class AVRO_DECL GenericMap : public GenericContainer { +public: + /** + * The contents type for the map. + */ + typedef std::vector > Value; + + /** + * Constructs a generic map corresponding to the given schema \p schema, + * which should be of Avro type map. + */ + GenericMap(const NodePtr& schema) : GenericContainer(AVRO_MAP, schema) { + } + + /** + * Returns the contents of this map. + */ + const Value& value() const { + return value_; + } + + /** + * Returns the reference to the contents of this map. + */ + Value& value() { + return value_; + } +private: + Value value_; +}; + +/** + * Generic container for Avro enum. + */ +class AVRO_DECL GenericEnum : public GenericContainer { + size_t value_; + + static size_t index(const NodePtr& schema, const std::string& symbol) { + size_t result; + if (schema->nameIndex(symbol, result)) { + return result; + } + throw Exception("No such symbol"); + } + +public: + /** + * Constructs a generic enum corresponding to the given schema \p schema, + * which should be of Avro type enum. + */ + GenericEnum(const NodePtr& schema) : + GenericContainer(AVRO_ENUM, schema), value_(0) { + } + + GenericEnum(const NodePtr& schema, const std::string& symbol) : + GenericContainer(AVRO_ENUM, schema), value_(index(schema, symbol)) { + } + + /** + * Returns the symbol corresponding to the cardinal \p n. If the + * value for \p n is not within the limits an exception is thrown. + */ + const std::string& symbol(size_t n) { + if (n < schema()->names()) { + return schema()->nameAt(n); + } + throw Exception("Not as many symbols"); + } + + /** + * Returns the cardinal for the given symbol \c symbol. If the symbol + * is not defined for this enum and exception is thrown. + */ + size_t index(const std::string& symbol) const { + return index(schema(), symbol); + } + + /** + * Set the value for this enum corresponding to the given symbol \c symbol. + */ + size_t set(const std::string& symbol) { + return value_ = index(symbol); + } + + /** + * Set the value for this enum corresponding to the given cardinal \c n. + */ + void set(size_t n) { + if (n < schema()->names()) { + value_ = n; + return; + } + throw Exception("Not as many symbols"); + } + + /** + * Returns the cardinal for the current value of this enum. + */ + size_t value() const { + return value_; + } + + /** + * Returns the symbol for the current value of this enum. + */ + const std::string& symbol() const { + return schema()->nameAt(value_); + } +}; + +/** + * Generic container for Avro fixed. + */ +class AVRO_DECL GenericFixed : public GenericContainer { + std::vector value_; +public: + /** + * Constructs a generic enum corresponding to the given schema \p schema, + * which should be of Avro type fixed. + */ + GenericFixed(const NodePtr& schema) : GenericContainer(AVRO_FIXED, schema) { + value_.resize(schema->fixedSize()); + } + + GenericFixed(const NodePtr& schema, const std::vector& v) : + GenericContainer(AVRO_FIXED, schema), value_(v) { } + + /** + * Returns the contents of this fixed. + */ + const std::vector& value() const { + return value_; + } + + /** + * Returns the reference to the contents of this fixed. + */ + std::vector& value() { + return value_; + } +}; + +inline size_t GenericDatum::unionBranch() const { + return boost::any_cast(&value_)->currentBranch(); +} + +inline void GenericDatum::selectBranch(size_t branch) { + boost::any_cast(&value_)->selectBranch(branch); +} + +} // namespace avro +#endif // avro_GenericDatum_hh__ diff --git a/lang/c++/api/Node.hh b/lang/c++/api/Node.hh index 3a9ec45d..ff227b9e 100644 --- a/lang/c++/api/Node.hh +++ b/lang/c++/api/Node.hh @@ -32,6 +32,7 @@ namespace avro { class Node; +class GenericDatum; typedef boost::shared_ptr NodePtr; @@ -122,6 +123,9 @@ class AVRO_DECL Node : private boost::noncopyable } virtual size_t leaves() const = 0; virtual const NodePtr& leafAt(int index) const = 0; + virtual const GenericDatum& defaultValueAt(int index) { + throw Exception(boost::format("No default value at: %1%") % index); + } void addName(const std::string &name) { checkLock(); diff --git a/lang/c++/api/NodeImpl.hh b/lang/c++/api/NodeImpl.hh index 0e75e91d..cbfcfb51 100644 --- a/lang/c++/api/NodeImpl.hh +++ b/lang/c++/api/NodeImpl.hh @@ -20,6 +20,7 @@ #define avro_NodeImpl_hh__ #include "Config.hh" +#include "GenericDatum.hh" #include #include @@ -261,26 +262,27 @@ class AVRO_DECL NodeSymbolic : public NodeImplSymbolic }; -class AVRO_DECL NodeRecord : public NodeImplRecord -{ - public: - - NodeRecord() : - NodeImplRecord(AVRO_RECORD) - { } - - NodeRecord(const HasName &name, const MultiLeaves &fields, const LeafNames &fieldsNames) : - NodeImplRecord(AVRO_RECORD, name, fields, fieldsNames, NoSize()) - { - for(size_t i=0; i < leafNameAttributes_.size(); ++i) { - if(!nameIndex_.add(leafNameAttributes_.get(i), i)) { - throw Exception(boost::format("Cannot add duplicate name: %1%") % leafNameAttributes_.get(i)); +class AVRO_DECL NodeRecord : public NodeImplRecord { + std::vector defaultValues; +public: + NodeRecord() : NodeImplRecord(AVRO_RECORD) { } + NodeRecord(const HasName &name, const MultiLeaves &fields, + const LeafNames &fieldsNames, + const std::vector& dv) : + NodeImplRecord(AVRO_RECORD, name, fields, fieldsNames, NoSize()), + defaultValues(dv) { + for (size_t i = 0; i < leafNameAttributes_.size(); ++i) { + if (!nameIndex_.add(leafNameAttributes_.get(i), i)) { + throw Exception(boost::format( + "Cannot add duplicate name: %1%") % + leafNameAttributes_.get(i)); } } } void swap(NodeRecord& r) { NodeImplRecord::swap(r); + defaultValues.swap(r.defaultValues); } SchemaResolution resolve(const Node &reader) const; @@ -288,10 +290,12 @@ class AVRO_DECL NodeRecord : public NodeImplRecord void printJson(std::ostream &os, int depth) const; bool isValid() const { - return ( - (nameAttribute_.size() == 1) && - (leafAttributes_.size() == leafNameAttributes_.size()) - ); + return ((nameAttribute_.size() == 1) && + (leafAttributes_.size() == leafNameAttributes_.size())); + } + + const GenericDatum& defaultValueAt(int index) { + return defaultValues[index]; } }; diff --git a/lang/c++/api/Stream.hh b/lang/c++/api/Stream.hh index ed88d76d..92b2334d 100644 --- a/lang/c++/api/Stream.hh +++ b/lang/c++/api/Stream.hh @@ -140,6 +140,13 @@ AVRO_DECL std::auto_ptr memoryInputStream(const uint8_t* data, size */ AVRO_DECL std::auto_ptr memoryInputStream(const OutputStream& source); +/** + * Returns the contents written so far into the output stream, which should + * be a memory output stream. That is it must have been returned by a pervious + * call to memoryOutputStream(). + */ +AVRO_DECL boost::shared_ptr > snapshot(const OutputStream& source); + /** * Returns a new OutputStream whose contents would be stored in a file. * Data is written in chunks of given buffer size. diff --git a/lang/c++/build.sh b/lang/c++/build.sh index 069af6b2..f0e41469 100755 --- a/lang/c++/build.sh +++ b/lang/c++/build.sh @@ -79,7 +79,6 @@ case "$target" in (cd build && make && cd .. \ && ./build/buffertest \ && ./build/unittest \ - && ./build/testgentest \ && ./build/CodecTests \ && ./build/StreamTests \ && ./build/SpecificTests \ diff --git a/lang/c++/examples/cpx.hh b/lang/c++/examples/cpx.hh index 2db2a811..cb83aa64 100644 --- a/lang/c++/examples/cpx.hh +++ b/lang/c++/examples/cpx.hh @@ -21,7 +21,6 @@ #define CPX_HH_1278398428__H_ -#include "boost/any.hpp" #include "avro/Specific.hh" #include "avro/Encoder.hh" #include "avro/Decoder.hh" diff --git a/lang/c++/impl/Compiler.cc b/lang/c++/impl/Compiler.cc index c61b2537..01bbde2d 100644 --- a/lang/c++/impl/Compiler.cc +++ b/lang/c++/impl/Compiler.cc @@ -25,19 +25,20 @@ #include "json/JsonDom.hh" -extern void yyparse(void *ctx); - using std::string; using std::map; using std::vector; +using std::pair; +using std::make_pair; namespace avro { +using json::Entity; using json::Object; using json::Array; +using json::EntityType; typedef map SymbolTable; -using json::Entity; // #define DEBUG_VERBOSE @@ -92,7 +93,7 @@ static NodePtr makeNode(const std::string& t, SymbolTable& st, const string& ns) } Name n = getName(t, ns); - map::const_iterator it = st.find(n); + SymbolTable::const_iterator it = st.find(n); if (it != st.end()) { return NodePtr(new NodeSymbolic(asSingleAttribute(n), it->second)); } @@ -145,16 +146,189 @@ const int64_t getLongField(const Entity& e, const Object& m, struct Field { const string& name; - const NodePtr value; - Field(const string& n, const NodePtr& v) : name(n), value(v) { } + const NodePtr schema; + const GenericDatum defaultValue; + Field(const string& n, const NodePtr& v, GenericDatum dv) : + name(n), schema(v), defaultValue(dv) { } }; +static void assertType(const Entity& e, EntityType et) +{ + if (e.type() != et) { + throw Exception(boost::format("Unexpected type for default value: " + "Expected %1%, but found %2%") % et % e.type()); + } +} + +static vector toBin(const std::string& s) +{ + vector result; + result.resize(s.size()); + std::copy(s.c_str(), s.c_str() + s.size(), &result[0]); + return result; +} + +static string nameof(const NodePtr& n) +{ + Type t = n->type(); + switch (t) { + case AVRO_STRING: + return "string"; + case AVRO_BYTES: + return "bytes"; + case AVRO_INT: + return "int"; + case AVRO_LONG: + return "long"; + case AVRO_FLOAT: + return "float"; + case AVRO_DOUBLE: + return "double"; + case AVRO_BOOL: + return "boolean"; + case AVRO_NULL: + return "null"; + case AVRO_RECORD: + case AVRO_ENUM: + case AVRO_FIXED: + case AVRO_SYMBOLIC: + return n->name().fullname(); + case AVRO_ARRAY: + return "array"; + case AVRO_MAP: + return "map"; + case AVRO_UNION: + return "union"; + default: + throw Exception(boost::format("Unknown type: %1%") % t); + } +} + +static GenericDatum makeGenericDatum(NodePtr n, const Entity& e, + const SymbolTable& st) +{ + Type t = n->type(); + if (t == AVRO_SYMBOLIC) { + n = st.find(n->name())->second; + t = n->type(); + } + switch (t) { + case AVRO_STRING: + assertType(e, json::etString); + return GenericDatum(e.stringValue()); + case AVRO_BYTES: + assertType(e, json::etString); + return GenericDatum(toBin(e.stringValue())); + case AVRO_INT: + assertType(e, json::etLong); + return GenericDatum(static_cast(e.longValue())); + case AVRO_LONG: + assertType(e, json::etLong); + return GenericDatum(e.longValue()); + case AVRO_FLOAT: + assertType(e, json::etDouble); + return GenericDatum(static_cast(e.doubleValue())); + case AVRO_DOUBLE: + assertType(e, json::etDouble); + return GenericDatum(e.doubleValue()); + case AVRO_BOOL: + assertType(e, json::etBool); + return GenericDatum(e.boolValue()); + case AVRO_NULL: + assertType(e, json::etNull); + return GenericDatum(); + case AVRO_RECORD: + { + assertType(e, json::etObject); + GenericRecord result(n); + const map& v = e.objectValue(); + for (size_t i = 0; i < n->leaves(); ++i) { + map::const_iterator it = v.find(n->nameAt(i)); + if (it == v.end()) { + throw Exception(boost::format( + "No value found in default for %1%") % n->nameAt(i)); + } + result.setFieldAt(i, + makeGenericDatum(n->leafAt(i), it->second, st)); + } + return GenericDatum(n, result); + } + case AVRO_ENUM: + assertType(e, json::etString); + return GenericDatum(n, GenericEnum(n, e.stringValue())); + case AVRO_ARRAY: + { + assertType(e, json::etArray); + GenericArray result(n); + const vector& elements = e.arrayValue(); + for (vector::const_iterator it = elements.begin(); + it != elements.end(); ++it) { + result.value().push_back(makeGenericDatum(n->leafAt(0), *it, st)); + } + return GenericDatum(n, result); + } + case AVRO_MAP: + { + assertType(e, json::etObject); + GenericMap result(n); + const map& v = e.objectValue(); + for (map::const_iterator it = v.begin(); + it != v.end(); ++it) { + result.value().push_back(make_pair(it->first, + makeGenericDatum(n->leafAt(1), it->second, st))); + } + return GenericDatum(n, result); + } + case AVRO_UNION: + { + GenericUnion result(n); + string name; + Entity e2; + if (e.type() == json::etNull) { + name = "null"; + e2 = e; + } else { + assertType(e, json::etObject); + const map& v = e.objectValue(); + if (v.size() != 1) { + throw Exception(boost::format("Default value for " + "union has more than one field: %1%") % e.toString()); + } + map::const_iterator it = v.begin(); + name = it->first; + e2 = it->second; + } + for (size_t i = 0; i < n->leaves(); ++i) { + const NodePtr& b = n->leafAt(i); + if (nameof(b) == name) { + result.selectBranch(i); + result.datum() = makeGenericDatum(b, e2, st); + return GenericDatum(n, result); + } + } + throw Exception(boost::format("Invalid default value %1%") % + e.toString()); + } + case AVRO_FIXED: + assertType(e, json::etString); + return GenericDatum(n, GenericFixed(n, toBin(e.stringValue()))); + default: + throw Exception(boost::format("Unknown type: %1%") % t); + } + return GenericDatum(); +} + + static Field makeField(const Entity& e, SymbolTable& st, const string& ns) { const Object& m = e.objectValue(); const string& n = getStringField(e, m, "name"); Object::const_iterator it = findField(e, m, "type"); - return Field(n, makeNode(it->second, st, ns)); + map::const_iterator it2 = m.find("default"); + NodePtr node = makeNode(it->second, st, ns); + GenericDatum d = (it2 == m.end()) ? GenericDatum() : + makeGenericDatum(node, it2->second, st); + return Field(n, node, d); } static NodePtr makeRecordNode(const Entity& e, @@ -163,14 +337,16 @@ static NodePtr makeRecordNode(const Entity& e, const Array& v = getArrayField(e, m, "fields"); concepts::MultiAttribute fieldNames; concepts::MultiAttribute fieldValues; + vector defaultValues; for (Array::const_iterator it = v.begin(); it != v.end(); ++it) { Field f = makeField(*it, st, ns); fieldNames.add(f.name); - fieldValues.add(f.value); + fieldValues.add(f.schema); + defaultValues.push_back(f.defaultValue); } return NodePtr(new NodeRecord(asSingleAttribute(name), - fieldValues, fieldNames)); + fieldValues, fieldNames, defaultValues)); } static NodePtr makeEnumNode(const Entity& e, @@ -302,6 +478,12 @@ AVRO_DECL ValidSchema compileJsonSchemaFromStream(InputStream& is) return ValidSchema(n); } +AVRO_DECL ValidSchema compileJsonSchemaFromFile(const char* filename) +{ + std::auto_ptr s = fileInputStream(filename); + return compileJsonSchemaFromStream(*s); +} + AVRO_DECL ValidSchema compileJsonSchemaFromMemory(const uint8_t* input, size_t len) { return compileJsonSchemaFromStream(*memoryInputStream(input, len)); diff --git a/lang/c++/impl/Generic.cc b/lang/c++/impl/Generic.cc index eb540e06..884fadb3 100644 --- a/lang/c++/impl/Generic.cc +++ b/lang/c++/impl/Generic.cc @@ -17,7 +17,6 @@ */ #include "Generic.hh" -#include "NodeImpl.hh" #include namespace avro { @@ -35,80 +34,6 @@ void GenericContainer::assertType(const NodePtr& schema, Type type) { } } -GenericDatum::GenericDatum(const ValidSchema& schema) : - type_(schema.root()->type()) -{ - init(schema.root()); -} - -GenericDatum::GenericDatum(const NodePtr& schema) : type_(schema->type()) -{ - init(schema); -} - -void GenericDatum::init(const NodePtr& schema) -{ - NodePtr sc = schema; - if (type_ == AVRO_SYMBOLIC) { - sc = resolveSymbol(schema); - type_ = sc->type(); - } - switch (type_) { - case AVRO_NULL: - break; - case AVRO_BOOL: - value_ = bool(); - break; - case AVRO_INT: - value_ = int32_t(); - break; - case AVRO_LONG: - value_ = int64_t(); - break; - case AVRO_FLOAT: - value_ = float(); - break; - case AVRO_DOUBLE: - value_ = double(); - break; - case AVRO_STRING: - value_ = string(); - break; - case AVRO_BYTES: - value_ = vector(); - break; - case AVRO_FIXED: - value_ = GenericFixed(sc); - break; - case AVRO_RECORD: - value_ = GenericRecord(sc); - break; - case AVRO_ENUM: - value_ = GenericEnum(sc); - break; - case AVRO_ARRAY: - value_ = GenericArray(sc); - break; - case AVRO_MAP: - value_ = GenericMap(sc); - break; - case AVRO_UNION: - value_ = GenericUnion(sc); - break; - default: - throw Exception(boost::format("Unknown schema type %1%") % - toString(type_)); - } -} - -GenericRecord::GenericRecord(const NodePtr& schema) : - GenericContainer(AVRO_RECORD, schema) { - fields_.resize(schema->leaves()); - for (size_t i = 0; i < schema->leaves(); ++i) { - fields_[i] = GenericDatum(schema->leafAt(i)); - } -} - GenericReader::GenericReader(const ValidSchema& s, const DecoderPtr& decoder) : schema_(s), isResolving_(dynamic_cast(&(*decoder)) != 0), decoder_(decoder) @@ -133,6 +58,8 @@ void GenericReader::read(GenericDatum& datum, Decoder& d, bool isResolving) { if (datum.isUnion()) { datum.selectBranch(d.decodeUnionIndex()); + read(datum.value().datum(), d, isResolving); + return; } switch (datum.type()) { case AVRO_NULL: @@ -249,6 +176,8 @@ void GenericWriter::write(const GenericDatum& datum, Encoder& e) { if (datum.isUnion()) { e.encodeUnionIndex(datum.unionBranch()); + write(datum.value().datum(), e); + return; } switch (datum.type()) { case AVRO_NULL: diff --git a/lang/c++/impl/GenericDatum.cc b/lang/c++/impl/GenericDatum.cc new file mode 100644 index 00000000..b5998a84 --- /dev/null +++ b/lang/c++/impl/GenericDatum.cc @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "GenericDatum.hh" +#include "NodeImpl.hh" + +using std::string; +using std::vector; + +namespace avro { + +GenericDatum::GenericDatum(const ValidSchema& schema) : + type_(schema.root()->type()) +{ + init(schema.root()); +} + +GenericDatum::GenericDatum(const NodePtr& schema) : type_(schema->type()) +{ + init(schema); +} + +void GenericDatum::init(const NodePtr& schema) +{ + NodePtr sc = schema; + if (type_ == AVRO_SYMBOLIC) { + sc = resolveSymbol(schema); + type_ = sc->type(); + } + switch (type_) { + case AVRO_NULL: + break; + case AVRO_BOOL: + value_ = bool(); + break; + case AVRO_INT: + value_ = int32_t(); + break; + case AVRO_LONG: + value_ = int64_t(); + break; + case AVRO_FLOAT: + value_ = float(); + break; + case AVRO_DOUBLE: + value_ = double(); + break; + case AVRO_STRING: + value_ = string(); + break; + case AVRO_BYTES: + value_ = vector(); + break; + case AVRO_FIXED: + value_ = GenericFixed(sc); + break; + case AVRO_RECORD: + value_ = GenericRecord(sc); + break; + case AVRO_ENUM: + value_ = GenericEnum(sc); + break; + case AVRO_ARRAY: + value_ = GenericArray(sc); + break; + case AVRO_MAP: + value_ = GenericMap(sc); + break; + case AVRO_UNION: + value_ = GenericUnion(sc); + break; + default: + throw Exception(boost::format("Unknown schema type %1%") % + toString(type_)); + } +} + +GenericRecord::GenericRecord(const NodePtr& schema) : + GenericContainer(AVRO_RECORD, schema) { + fields_.resize(schema->leaves()); + for (size_t i = 0; i < schema->leaves(); ++i) { + fields_[i] = GenericDatum(schema->leafAt(i)); + } +} + +} // namespace avro diff --git a/lang/c++/impl/NodeImpl.cc b/lang/c++/impl/NodeImpl.cc index 0215f221..aba2a736 100644 --- a/lang/c++/impl/NodeImpl.cc +++ b/lang/c++/impl/NodeImpl.cc @@ -274,8 +274,8 @@ NodeFixed::printJson(std::ostream &os, int depth) const { os << "{\n"; os << indent(++depth) << "\"type\": \"fixed\",\n"; - os << indent(depth) << "\"size\": " << sizeAttribute_.get() << ",\n"; printName(os, nameAttribute_.get(), depth); + os << indent(depth) << "\"size\": " << sizeAttribute_.get() << "\n"; os << indent(--depth) << '}'; } diff --git a/lang/c++/impl/Stream.cc b/lang/c++/impl/Stream.cc index 944d69de..5da5edbc 100644 --- a/lang/c++/impl/Stream.cc +++ b/lang/c++/impl/Stream.cc @@ -21,6 +21,8 @@ namespace avro { +using std::vector; + class MemoryInputStream : public InputStream { const std::vector& data_; const size_t chunkSize_; @@ -176,5 +178,21 @@ std::auto_ptr memoryInputStream(const OutputStream& source) (mos.chunkSize_ - mos.available_))); } +boost::shared_ptr > snapshot(const OutputStream& source) +{ + const MemoryOutputStream& mos = + dynamic_cast(source); + boost::shared_ptr > result(new std::vector()); + size_t c = mos.byteCount_; + result->reserve(mos.byteCount_); + for (vector::const_iterator it = mos.data_.begin(); + it != mos.data_.end(); ++it) { + size_t n = std::min(c, mos.chunkSize_); + std::copy(*it, *it + n, std::back_inserter(*result)); + c -= n; + } + return result; +} + } // namespace avro diff --git a/lang/c++/impl/avrogencpp.cc b/lang/c++/impl/avrogencpp.cc index 3e8c4749..a6a858d5 100644 --- a/lang/c++/impl/avrogencpp.cc +++ b/lang/c++/impl/avrogencpp.cc @@ -533,10 +533,27 @@ void CodeGen::generateRecordTraits(const NodePtr& n) os_ << " }\n" << " static void decode(Decoder& d, " << fn << "& v) {\n"; + os_ << " if (avro::ResolvingDecoder *rd =\n"; + os_ << " dynamic_cast(&d)) {\n"; + os_ << " const std::vector fo = rd->fieldOrder();\n"; + os_ << " for (std::vector::const_iterator it = fo.begin();\n"; + os_ << " it != fo.end(); ++it) {\n"; + os_ << " switch (*it) {\n"; + for (size_t i = 0; i < c; ++i) { + os_ << " case " << i << ":\n"; + os_ << " avro::decode(d, v." << n->nameAt(i) << ");\n"; + os_ << " break;\n"; + } + os_ << " default:\n"; + os_ << " break;\n"; + os_ << " }\n"; + os_ << " }\n"; + os_ << " } else {\n"; for (size_t i = 0; i < c; ++i) { - os_ << " avro::decode(d, v." << n->nameAt(i) << ");\n"; + os_ << " avro::decode(d, v." << n->nameAt(i) << ");\n"; } + os_ << " }\n"; os_ << " }\n" << "};\n\n"; diff --git a/lang/c++/impl/parsing/ResolvingDecoder.cc b/lang/c++/impl/parsing/ResolvingDecoder.cc index 9443e271..e0d25edf 100644 --- a/lang/c++/impl/parsing/ResolvingDecoder.cc +++ b/lang/c++/impl/parsing/ResolvingDecoder.cc @@ -36,6 +36,8 @@ #include "Decoder.hh" #include "Encoder.hh" #include "NodeImpl.hh" +#include "Generic.hh" +#include "Stream.hh" namespace avro { @@ -45,7 +47,9 @@ namespace parsing { using boost::shared_ptr; using boost::static_pointer_cast; +using boost::make_shared; +using std::auto_ptr; using std::map; using std::pair; using std::vector; @@ -62,13 +66,13 @@ typedef pair NodePair; class ResolvingGrammarGenerator : public ValidatingGrammarGenerator { ProductionPtr doGenerate2(const NodePtr& writer, const NodePtr& reader, map &m, - const map &m2); + map &m2); ProductionPtr resolveRecords(const NodePtr& writer, const NodePtr& reader, map &m, - const map &m2); + map &m2); ProductionPtr resolveUnion(const NodePtr& writer, const NodePtr& reader, map &m, - const map &m2); + map &m2); static vector > fields(const NodePtr& n) { vector > result; @@ -82,7 +86,7 @@ class ResolvingGrammarGenerator : public ValidatingGrammarGenerator { static int bestBranch(const NodePtr& writer, const NodePtr& reader); ProductionPtr getWriterProduction(const NodePtr& n, - const map& m2); + map& m2); public: Symbol generate( @@ -99,7 +103,7 @@ Symbol ResolvingGrammarGenerator::generate( fixup(backup, m2); map m; - ProductionPtr main = doGenerate2(rr, rw, m, m2); + ProductionPtr main = doGenerate2(rw, rr, m, m2); fixup(main, m); return Symbol::rootSymbol(main, backup); } @@ -111,7 +115,10 @@ int ResolvingGrammarGenerator::bestBranch(const NodePtr& writer, const size_t c = reader->leaves(); for (size_t j = 0; j < c; ++j) { - const NodePtr& r = reader->leafAt(j); + NodePtr r = reader->leafAt(j); + if (r->type() == AVRO_SYMBOLIC) { + r = resolveSymbol(r); + } if (t == r->type()) { if (r->hasName()) { if (r->name() == writer->name()) { @@ -145,6 +152,17 @@ int ResolvingGrammarGenerator::bestBranch(const NodePtr& writer, return -1; } +static shared_ptr > getAvroBinary( + const GenericDatum& defaultValue) +{ + EncoderPtr e = binaryEncoder(); + auto_ptr os = memoryOutputStream(); + e->init(*os); + GenericWriter::write(*e, defaultValue); + e->flush(); + return snapshot(*os); +} + template struct equalsFirst { @@ -156,19 +174,24 @@ struct equalsFirst }; ProductionPtr ResolvingGrammarGenerator::getWriterProduction( - const NodePtr& n, const map& m2) + const NodePtr& n, map& m2) { const NodePtr& nn = (n->type() == AVRO_SYMBOLIC) ? static_cast(*n).getNode() : n; map::const_iterator it2 = m2.find(nn); - return (it2 != m2.end()) ? it2->second : - ValidatingGrammarGenerator::generate(nn); + if (it2 != m2.end()) { + return it2->second; + } else { + ProductionPtr result = ValidatingGrammarGenerator::doGenerate(nn, m2); + fixup(result, m2); + return result; + } } ProductionPtr ResolvingGrammarGenerator::resolveRecords( const NodePtr& writer, const NodePtr& reader, map& m, - const map& m2) + map& m2) { ProductionPtr result = make_shared(); @@ -177,6 +200,12 @@ ProductionPtr ResolvingGrammarGenerator::resolveRecords( vector fieldOrder; fieldOrder.reserve(reader->names()); + /* + * We look for all writer fields in the reader. If found, recursively + * resolve the corresponding fields. Then erase the reader field. + * If no matching field is found for reader, arrange to skip the writer + * field. + */ for (vector >::const_iterator it = wf.begin(); it != wf.end(); ++it) { vector >::iterator it2 = @@ -190,18 +219,38 @@ ProductionPtr ResolvingGrammarGenerator::resolveRecords( rf.erase(it2); } else { ProductionPtr p = getWriterProduction( - writer->leafAt(it->second), m2); + writer->leafAt(it->second), m2); result->push_back(Symbol::skipStart()); if (p->size() == 1) { result->push_back((*p)[0]); } else { result->push_back(Symbol::indirect(p)); - } + } } } - if (! rf.empty()) { - throw Exception("Don't know how to handle excess fields for reader."); + /* + * Examine the reader fields left out, (i.e. those didn't have corresponding + * writer field). + */ + for (vector >::const_iterator it = rf.begin(); + it != rf.end(); ++it) { + + NodePtr s = reader->leafAt(it->second); + fieldOrder.push_back(it->second); + + if (s->type() == AVRO_SYMBOLIC) { + s = resolveSymbol(s); + } + shared_ptr > defaultBinary = + getAvroBinary(reader->defaultValueAt(it->second)); + result->push_back(Symbol::defaultStartAction(defaultBinary)); + map >::const_iterator it2 = + m.find(NodePair(s, s)); + ProductionPtr p = (it2 == m.end()) ? + doGenerate2(s, s, m, m2) : it2->second; + copy(p->rbegin(), p->rend(), back_inserter(*result)); + result->push_back(Symbol::defaultEndAction()); } reverse(result->begin(), result->end()); result->push_back(Symbol::sizeListAction(fieldOrder)); @@ -214,7 +263,7 @@ ProductionPtr ResolvingGrammarGenerator::resolveRecords( ProductionPtr ResolvingGrammarGenerator::resolveUnion( const NodePtr& writer, const NodePtr& reader, map& m, - const map& m2) + map& m2) { vector v; size_t c = writer->leaves(); @@ -230,10 +279,12 @@ ProductionPtr ResolvingGrammarGenerator::resolveUnion( } ProductionPtr ResolvingGrammarGenerator::doGenerate2( - const NodePtr& writer, const NodePtr& reader, + const NodePtr& w, const NodePtr& r, map &m, - const map &m2) + map &m2) { + const NodePtr writer = w->type() == AVRO_SYMBOLIC ? resolveSymbol(w) : w; + const NodePtr reader = r->type() == AVRO_SYMBOLIC ? resolveSymbol(r) : r; Type writerType = writer->type(); Type readerType = reader->type(); @@ -268,7 +319,12 @@ ProductionPtr ResolvingGrammarGenerator::doGenerate2( case AVRO_RECORD: if (writer->name() == reader->name()) { const pair key(writer, reader); - m.erase(key); + map::const_iterator kp = m.find(key); + if (kp != m.end()) { + return (kp->second) ? kp->second : + make_shared(1, Symbol::placeholder(key)); + } + m[key] = ProductionPtr(); ProductionPtr result = resolveRecords(writer, reader, m, m2); m[key] = result; return result; @@ -297,11 +353,14 @@ ProductionPtr ResolvingGrammarGenerator::doGenerate2( } case AVRO_MAP: { - ProductionPtr v = doGenerate2(writer->leafAt(1), - reader->leafAt(1), m, m2); + ProductionPtr pp = + doGenerate2(writer->leafAt(1),reader->leafAt(1), m, m2); + ProductionPtr v(new Production(*pp)); v->push_back(Symbol::stringSymbol()); - ProductionPtr v2 = getWriterProduction(writer->leafAt(1), m2); + ProductionPtr pp2 = getWriterProduction(writer->leafAt(1), m2); + ProductionPtr v2(new Production(*pp2)); + v2->push_back(Symbol::stringSymbol()); ProductionPtr result = make_shared(); @@ -387,23 +446,48 @@ ProductionPtr ResolvingGrammarGenerator::doGenerate2( } class ResolvingDecoderHandler { - Decoder& base_; -public: - ResolvingDecoderHandler(Decoder& base) : base_(base) { } + shared_ptr > defaultData_; + auto_ptr inp_; + DecoderPtr backup_; + DecoderPtr& base_; + const DecoderPtr binDecoder; + public: + ResolvingDecoderHandler(DecoderPtr& base) : base_(base), + binDecoder(binaryDecoder()) { } size_t handle(const Symbol& s) { switch (s.kind()) { case Symbol::sWriterUnion: - return base_.decodeUnionIndex(); + return base_->decodeUnionIndex(); + case Symbol::sDefaultStart: + defaultData_ = s.extra > >(); + backup_ = base_; + inp_ = memoryInputStream(&(*defaultData_)[0], defaultData_->size()); + base_ = binDecoder; + base_->init(*inp_); + return 0; + case Symbol::sDefaultEnd: + base_= backup_; + backup_.reset(); + return 0; default: return 0; } } + + void reset() + { + if (backup_ != NULL) + { + base_= backup_; + backup_.reset(); + } + } }; template class ResolvingDecoderImpl : public ResolvingDecoder { - const DecoderPtr base_; + DecoderPtr base_; ResolvingDecoderHandler handler_; Parser parser_; @@ -433,8 +517,8 @@ class ResolvingDecoderImpl : public ResolvingDecoder ResolvingDecoderImpl(const ValidSchema& writer, const ValidSchema& reader, const DecoderPtr& base) : base_(base), - handler_(*base_), - parser_(ResolvingGrammarGenerator().generate(reader, writer), + handler_(base_), + parser_(ResolvingGrammarGenerator().generate(writer, reader), &(*base_), handler_) { } @@ -443,6 +527,7 @@ class ResolvingDecoderImpl : public ResolvingDecoder template void ResolvingDecoderImpl

    ::init(InputStream& is) { + handler_.reset(); base_->init(is); parser_.reset(); } @@ -563,6 +648,7 @@ size_t ResolvingDecoderImpl

    ::arrayStart() template size_t ResolvingDecoderImpl

    ::arrayNext() { + parser_.processImplicitActions(); size_t result = base_->arrayNext(); if (result == 0) { parser_.popRepeater(); @@ -605,6 +691,7 @@ size_t ResolvingDecoderImpl

    ::mapStart() template size_t ResolvingDecoderImpl

    ::mapNext() { + parser_.processImplicitActions(); size_t result = base_->mapNext(); if (result == 0) { parser_.popRepeater(); diff --git a/lang/c++/impl/parsing/Symbol.cc b/lang/c++/impl/parsing/Symbol.cc index 4acc0098..a651f401 100644 --- a/lang/c++/impl/parsing/Symbol.cc +++ b/lang/c++/impl/parsing/Symbol.cc @@ -63,6 +63,8 @@ const char* Symbol::stringValues[] = { "Record", "SizeList", "WriterUnion", + "DefaultStart", + "DefaultEnd", "ImplicitActionHigh", "Error" }; @@ -74,7 +76,6 @@ Symbol Symbol::enumAdjustSymbol(const NodePtr& writer, const NodePtr& reader) for (size_t i = 0; i < rc; ++i) { rs.push_back(reader->nameAt(i)); } - sort(rs.begin(), rs.end()); size_t wc = writer->names(); vector adj; diff --git a/lang/c++/impl/parsing/Symbol.hh b/lang/c++/impl/parsing/Symbol.hh index ec6d3be8..a7c09975 100644 --- a/lang/c++/impl/parsing/Symbol.hh +++ b/lang/c++/impl/parsing/Symbol.hh @@ -85,6 +85,8 @@ public: sRecord, sSizeList, sWriterUnion, + sDefaultStart, // extra has default value in Avro binary encoding + sDefaultEnd, sImplicitActionHigh, sError }; @@ -215,6 +217,16 @@ public: return Symbol(sRepeater, boost::make_tuple(s, isArray, read, skip)); } + static Symbol defaultStartAction(boost::shared_ptr > bb) + { + return Symbol(sDefaultStart, bb); + } + + static Symbol defaultEndAction() + { + return Symbol(sDefaultEnd); + } + static Symbol alternative( const std::vector& branches) { @@ -287,6 +299,10 @@ public: }; +/** + * Recursively replaces all placeholders in the production with the + * corresponding values. + */ template void fixup(const ProductionPtr& p, const std::map &m) @@ -298,6 +314,10 @@ void fixup(const ProductionPtr& p, } +/** + * Recursively replaces all placeholders in the symbol with the values with the + * corresponding values. + */ template void fixup_internal(const ProductionPtr& p, const std::map &m, @@ -335,11 +355,16 @@ void fixup(Symbol& s, const std::map &m, fixup_internal(boost::tuples::get<2>(ri), m, seen); fixup_internal(boost::tuples::get<3>(ri), m, seen); } - break; case Symbol::sPlaceholder: - s = Symbol::symbolic(boost::weak_ptr( - m.find(s.extra())->second)); + { + typename std::map >::const_iterator it = + m.find(s.extra()); + if (it == m.end()) { + throw Exception("Placeholder symbol cannot be resolved"); + } + s = Symbol::symbolic(boost::weak_ptr(it->second)); + } break; case Symbol::sUnionAdjust: fixup_internal(s.extrap >()->second, @@ -450,11 +475,12 @@ public: break; default: if (s.isImplicitAction()) { - Symbol ss = s; - parsingStack.pop(); - size_t n = handler_.handle(ss); - if (ss.kind() == Symbol::sWriterUnion) { + size_t n = handler_.handle(s); + if (s.kind() == Symbol::sWriterUnion) { + parsingStack.pop(); selectBranch(n); + } else { + parsingStack.pop(); } } else { std::ostringstream oss; @@ -730,6 +756,42 @@ public: }; +inline std::ostream& operator<<(std::ostream& os, const Symbol s); + +inline std::ostream& operator<<(std::ostream& os, const Production p) +{ + os << '('; + for (Production::const_iterator it = p.begin(); it != p.end(); ++it) { + os << *it << ", "; + } + os << ')'; + return os; +} + +inline std::ostream& operator<<(std::ostream& os, const Symbol s) +{ + switch (s.kind()) { + case Symbol::sRepeater: + { + const RepeaterInfo& ri = *s.extrap(); + os << '(' << Symbol::toString(s.kind()) + << boost::tuples::get<2>(ri) + << boost::tuples::get<3>(ri) + << ')'; + } + break; + case Symbol::sIndirect: + { + os << '(' << Symbol::toString(s.kind()) << ' ' + << *s.extra >() << ')'; + } + break; + default: + os << Symbol::toString(s.kind()); + break; + } + return os; + } } // namespace parsing } // namespace avro diff --git a/lang/c++/impl/parsing/ValidatingCodec.cc b/lang/c++/impl/parsing/ValidatingCodec.cc index ee111e4a..46670e1f 100644 --- a/lang/c++/impl/parsing/ValidatingCodec.cc +++ b/lang/c++/impl/parsing/ValidatingCodec.cc @@ -123,7 +123,8 @@ ProductionPtr ValidatingGrammarGenerator::doGenerate(const NodePtr& n, } case AVRO_MAP: { - ProductionPtr v = doGenerate(n->leafAt(1), m); + ProductionPtr pp = doGenerate(n->leafAt(1), m); + ProductionPtr v(new Production(*pp)); v->push_back(Symbol::stringSymbol()); ProductionPtr result = make_shared(); result->push_back(Symbol::mapEndSymbol()); diff --git a/lang/c++/jsonschemas/bigrecord b/lang/c++/jsonschemas/bigrecord index 815a32da..02dbccb2 100644 --- a/lang/c++/jsonschemas/bigrecord +++ b/lang/c++/jsonschemas/bigrecord @@ -34,6 +34,13 @@ "values": "int" } }, + { + "name": "recordmap", + "type": { + "type": "map", + "values": "Nested" + } + }, { "name": "myarray", "type": { diff --git a/lang/c++/jsonschemas/bigrecord_r b/lang/c++/jsonschemas/bigrecord_r new file mode 100644 index 00000000..f079162d --- /dev/null +++ b/lang/c++/jsonschemas/bigrecord_r @@ -0,0 +1,166 @@ +{ + "type": "record", + "name": "RootRecord", + "fields": [ + { + "name": "mylong", + "type": "long" + }, + { + "name": "mymap", + "type": { + "type": "map", + "values": "int" + } + }, + { + "name": "nestedrecord", + "type": { + "type": "record", + "name": "Nested", + "fields": [ + { + "name": "inval2", + "type": "string" + }, + { + "name": "inval1", + "type": "double" + }, + { + "name": "inval3", + "type": "int" + } + ] + } + }, + { + "name": "recordmap", + "type": { + "type": "map", + "values": "Nested" + } + }, + { + "name": "withDefaultValue", + "type": { + "type": "record", + "name": "WithDefaultValue", + "fields": [ + { + "name": "s1", + "type": "string" + }, + { + "name": "d1", + "type": "double" + }, + { + "name": "i1", + "type": "int" + } + ] + }, + "default": { + "s1": "sval", + "d1": 5.67, + "i1": 99 + } + }, + { + "name": "union1WithDefaultValue", + "type": [ "string", "int" ], + "default": { + "string": "sval" + } + }, + { + "name": "union2WithDefaultValue", + "type": [ "string", "null" ], + "default": null + }, + { + "name": "myarray", + "type": { + "type": "array", + "items": "double" + } + }, + { + "name": "myenum", + "type": { + "type": "enum", + "name": "ExampleEnum", + "symbols": [ + "zero", + "one", + "two", + "three" + ] + } + }, + { + "name": "myunion", + "type": [ + "null", + { + "type": "map", + "values": "int" + }, + "float" + ], + "default": null + }, + { + "name": "anotherunion", + "type": [ + "bytes", + "null" + ] + }, + { + "name": "mybool", + "type": "boolean" + }, + { + "name": "anothernested", + "type": "Nested" + }, + { + "name": "rwd", + "type": { + "type": "record", + "name": "RecordWithDefault", + "fields": [ + { + "name": "rwd_f1", + "type": "Nested" + } + ] + }, + "default": { + "rwd_f1": { + "inval2": "hello", + "inval1": 4.23, + "inval3": 100 + } + } + }, + { + "name": "myfixed", + "type": { + "type": "fixed", + "size": 16, + "name": "md5" + } + }, + { + "name": "anotherint", + "type": "int" + }, + { + "name": "bytes", + "type": "bytes" + } + ] +} diff --git a/lang/c++/test/AvrogencppTests.cc b/lang/c++/test/AvrogencppTests.cc index 20f4aee8..2312676e 100644 --- a/lang/c++/test/AvrogencppTests.cc +++ b/lang/c++/test/AvrogencppTests.cc @@ -18,6 +18,7 @@ #include "empty_record.hh" #include "bigrecord.hh" +#include "bigrecord_r.hh" #include "bigrecord2.hh" #include "tweet.hh" #include "union_array_union.hh" @@ -60,7 +61,7 @@ using avro::validatingEncoder; using avro::binaryDecoder; using avro::validatingDecoder; -void setRecord(testgen::RootRecord &myRecord) +void setRecord(testgen::RootRecord &myRecord) { uint8_t fixed[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}; @@ -97,7 +98,8 @@ void setRecord(testgen::RootRecord &myRecord) myRecord.bytes.push_back(20); } -void check(const testgen::RootRecord& r1, const testgen::RootRecord& r2) +template +void checkRecord(const T1& r1, const T2& r2) { BOOST_CHECK_EQUAL(r1.mylong, r2.mylong); BOOST_CHECK_EQUAL(r1.nestedrecord.inval1, r2.nestedrecord.inval1); @@ -116,10 +118,20 @@ void check(const testgen::RootRecord& r1, const testgen::RootRecord& r2) BOOST_CHECK_EQUAL_COLLECTIONS(r1.myfixed.begin(), r1.myfixed.end(), r2.myfixed.begin(), r2.myfixed.end()); BOOST_CHECK_EQUAL(r1.anotherint, r2.anotherint); + BOOST_CHECK_EQUAL(r1.bytes.size(), r2.bytes.size()); BOOST_CHECK_EQUAL_COLLECTIONS(r1.bytes.begin(), r1.bytes.end(), r2.bytes.begin(), r2.bytes.end()); + BOOST_CHECK_EQUAL(r1.myenum, r2.myenum); } +void checkDefaultValues(const testgen_r::RootRecord& r) +{ + BOOST_CHECK_EQUAL(r.withDefaultValue.s1, "sval"); + BOOST_CHECK_EQUAL(r.withDefaultValue.i1, 99); + BOOST_CHECK_CLOSE(r.withDefaultValue.d1, 5.67, 1e-10); +} + + void testEncoding() { ValidSchema s; @@ -139,7 +151,43 @@ void testEncoding() testgen::RootRecord t2; avro::decode(*d, t2); - check(t2, t1); + checkRecord(t2, t1); +} + +void testResolution() +{ + ValidSchema s_w; + ifstream ifs_w("jsonschemas/bigrecord"); + compileJsonSchema(ifs_w, s_w); + auto_ptr os = memoryOutputStream(); + EncoderPtr e = validatingEncoder(s_w, binaryEncoder()); + e->init(*os); + testgen::RootRecord t1; + setRecord(t1); + avro::encode(*e, t1); + e->flush(); + + ValidSchema s_r; + ifstream ifs_r("jsonschemas/bigrecord_r"); + compileJsonSchema(ifs_r, s_r); + DecoderPtr dd = binaryDecoder(); + auto_ptr is = memoryInputStream(*os); + dd->init(*is); + DecoderPtr rd = resolvingDecoder(s_w, s_r, dd); + testgen_r::RootRecord t2; + avro::decode(*rd, t2); + + checkRecord(t2, t1); + checkDefaultValues(t2); + + //Re-use the resolving decoder to decode again. + auto_ptr is1 = memoryInputStream(*os); + rd->init(*is1); + testgen_r::RootRecord t3; + avro::decode(*rd, t3); + checkRecord(t3, t1); + checkDefaultValues(t3); + } void testNamespace() @@ -210,10 +258,11 @@ void testEncoding2() } boost::unit_test::test_suite* -init_unit_test_suite(int argc, char* argv[]) +init_unit_test_suite(int argc, char* argv[]) { boost::unit_test::test_suite* ts = BOOST_TEST_SUITE("Code generator tests"); ts->add(BOOST_TEST_CASE(testEncoding)); + ts->add(BOOST_TEST_CASE(testResolution)); ts->add(BOOST_TEST_CASE(testEncoding2)); ts->add(BOOST_TEST_CASE(testEncoding2)); ts->add(BOOST_TEST_CASE(testNamespace)); diff --git a/lang/c++/test/CodecTests.cc b/lang/c++/test/CodecTests.cc index bae0b398..61d29a29 100644 --- a/lang/c++/test/CodecTests.cc +++ b/lang/c++/test/CodecTests.cc @@ -352,6 +352,7 @@ static vector::const_iterator skipCalls(Scanner& sc, Decoder& d, break; case 's': case 'N': + case 'R': break; default: BOOST_FAIL("Don't know how to skip: " << c); @@ -754,6 +755,9 @@ template void testGeneric(const TestData& td) { static int testNo = 0; testNo++; + BOOST_TEST_CHECKPOINT("Test: " << testNo << ' ' + << " schema: " << td.schema + << " calls: " << td.calls); ValidSchema vs = makeValidSchema(td.schema); @@ -1183,7 +1187,6 @@ static const TestData4 data4[] = { "{\"name\":\"f1\", \"type\":\"long\"}]}", "RLS10", { "10", "hello", NULL }, 1 }, - /* // Default values { "{\"type\":\"record\",\"name\":\"r\",\"fields\":[]}", "", { NULL }, @@ -1215,16 +1218,44 @@ static const TestData4 data4[] = { // Default value for a record. { "{\"type\":\"record\",\"name\":\"outer\",\"fields\":[" - "{\"name\": \"g2\", \"type\": \"long\"}]}", "L", - { "11", NULL }, + "{\"name\": \"g1\", " + "\"type\":{\"type\":\"record\",\"name\":\"inner1\",\"fields\":[" + "{\"name\":\"f1\", \"type\":\"long\" }," + "{\"name\":\"f2\", \"type\":\"int\"}] } }, " + "{\"name\": \"g2\", \"type\": \"long\"}]}", "LIL", + { "10", "12", "13", NULL }, "{\"type\":\"record\",\"name\":\"outer\",\"fields\":[" "{\"name\": \"g1\", " - "\"type\":{\"type\":\"record\",\"name\":\"inner\",\"fields\":[" - "{\"name\":\"f1\", \"type\":\"int\" }," + "\"type\":{\"type\":\"record\",\"name\":\"inner1\",\"fields\":[" + "{\"name\":\"f1\", \"type\":\"long\" }," + "{\"name\":\"f2\", \"type\":\"int\"}] } }, " + "{\"name\": \"g2\", \"type\": \"long\"}," + "{\"name\": \"g3\", " + "\"type\":{\"type\":\"record\",\"name\":\"inner2\",\"fields\":[" + "{\"name\":\"f1\", \"type\":\"long\" }," "{\"name\":\"f2\", \"type\":\"int\"}] }, " - "\"default\": { \"f1\": 10, \"f2\": 101 } }, " - "{\"name\": \"g2\", \"type\": \"long\"}]}", "RLRII", - { "11", "10", "101", NULL}, 1 }, + "\"default\": { \"f1\": 15, \"f2\": 101 } }] } ", + "RRLILRLI", + { "10", "12", "13", "15", "101", NULL}, 1 }, + + { "{\"type\":\"record\",\"name\":\"outer\",\"fields\":[" + "{\"name\": \"g1\", " + "\"type\":{\"type\":\"record\",\"name\":\"inner1\",\"fields\":[" + "{\"name\":\"f1\", \"type\":\"long\" }," + "{\"name\":\"f2\", \"type\":\"int\"}] } }, " + "{\"name\": \"g2\", \"type\": \"long\"}]}", "LIL", + { "10", "12", "13", NULL }, + "{\"type\":\"record\",\"name\":\"outer\",\"fields\":[" + "{\"name\": \"g1\", " + "\"type\":{\"type\":\"record\",\"name\":\"inner1\",\"fields\":[" + "{\"name\":\"f1\", \"type\":\"long\" }," + "{\"name\":\"f2\", \"type\":\"int\"}] } }, " + "{\"name\": \"g2\", \"type\": \"long\"}," + "{\"name\": \"g3\", " + "\"type\":\"inner1\", " + "\"default\": { \"f1\": 15, \"f2\": 101 } }] } ", + "RRLILRLI", + { "10", "12", "13", "15", "101", NULL}, 1 }, { "{\"type\":\"record\",\"name\":\"r\",\"fields\":[]}", "", { NULL }, @@ -1234,14 +1265,14 @@ static const TestData4 data4[] = { { "100", NULL }, 1 }, { "{ \"type\": \"array\", \"items\": {\"type\":\"record\"," - "\"name\":\"r\",\"fields\":[]} }", "[c1s]", - { NULL }, + "\"name\":\"r\",\"fields\":[" + "{\"name\":\"f0\", \"type\": \"int\"}]} }", "[c1sI]", + { "99", NULL }, "{ \"type\": \"array\", \"items\": {\"type\":\"record\"," "\"name\":\"r\",\"fields\":[" "{\"name\":\"f\", \"type\":\"int\", \"default\": 100}]} }", - "[c1sI]", + "[Rc1sI]", { "100", NULL }, 1 }, - */ // Enum resolution { "{\"type\":\"enum\",\"name\":\"e\",\"symbols\":[\"x\",\"y\",\"z\"]}", diff --git a/lang/c++/test/DataFileTests.cc b/lang/c++/test/DataFileTests.cc index 4959e7cd..95e80b1b 100644 --- a/lang/c++/test/DataFileTests.cc +++ b/lang/c++/test/DataFileTests.cc @@ -95,6 +95,18 @@ template <> struct codec_traits { } }; +template<> struct codec_traits { + static void encode(Encoder& e, const uint32_t& v) { + e.encodeFixed( (uint8_t *) &v,sizeof(uint32_t)); + } + + static void decode(Decoder& d, uint32_t& v) { + std::vector value; + d.decodeFixed(sizeof(uint32_t),value); + memcpy(&v,&(value[0]),sizeof(uint32_t)); + } +}; + } static ValidSchema makeValidSchema(const char* schema) @@ -123,6 +135,8 @@ static const char dblsch[] = "{\"type\": \"record\"," "\"name\":\"ComplexDouble\", \"fields\": [" "{\"name\":\"re\", \"type\":\"double\"}" "]}"; +static const char fsch[] = "{\"type\": \"fixed\"," + "\"name\":\"Fixed_32\", \"size\":4}"; string toString(const ValidSchema& s) @@ -419,6 +433,21 @@ class DataFileTest { } } } + + void testSchemaReadWrite() { + uint32_t a=42; + { + avro::DataFileWriter df(filename, writerSchema); + df.write(a); + } + + { + avro::DataFileReader df(filename); + uint32_t b; + df.read(b); + BOOST_CHECK_EQUAL(b, a); + } + } }; void addReaderTests(test_suite* ts, const shared_ptr& t) @@ -464,6 +493,9 @@ init_unit_test_suite( int argc, char* argv[] ) shared_ptr t6(new DataFileTest("test6.df", dsch, dblsch)); ts->add(BOOST_CLASS_TEST_CASE(&DataFileTest::testZip, t6)); + shared_ptr t7(new DataFileTest("test7.df",fsch,fsch)); + ts->add(BOOST_CLASS_TEST_CASE(&DataFileTest::testSchemaReadWrite,t7)); + ts->add(BOOST_CLASS_TEST_CASE(&DataFileTest::testCleanup,t7)); return ts; } diff --git a/lang/csharp/Avro.sln b/lang/csharp/Avro.sln index 58687b26..4134b857 100644 --- a/lang/csharp/Avro.sln +++ b/lang/csharp/Avro.sln @@ -1,95 +1,102 @@ - -Microsoft Visual Studio Solution File, Format Version 11.00 -# Visual Studio 2010 -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.main", "src\apache\main\Avro.main.csproj", "{A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.test", "src\apache\test\Avro.test.csproj", "{911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.codegen", "src\apache\codegen\Avro.codegen.csproj", "{BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.ipc", "src\apache\ipc\Avro.ipc.csproj", "{3B05043A-DC6C-49B6-85BF-9AB055D0B414}" - ProjectSection(ProjectDependencies) = postProject - {AEB22F94-4ECF-4008-B159-389B3F05D54B} = {AEB22F94-4ECF-4008-B159-389B3F05D54B} - EndProjectSection -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.msbuild", "src\apache\msbuild\Avro.msbuild.csproj", "{AEB22F94-4ECF-4008-B159-389B3F05D54B}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.perf", "src\apache\perf\Avro.perf.csproj", "{AC4E1909-2594-4D01-9B2B-B832C07BAFE5}" -EndProject -Global - GlobalSection(SolutionConfigurationPlatforms) = preSolution - Debug|Any CPU = Debug|Any CPU - Debug|Mixed Platforms = Debug|Mixed Platforms - Debug|x86 = Debug|x86 - Release|Any CPU = Release|Any CPU - Release|Mixed Platforms = Release|Mixed Platforms - Release|x86 = Release|x86 - EndGlobalSection - GlobalSection(ProjectConfigurationPlatforms) = postSolution - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Any CPU.Build.0 = Debug|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|x86.ActiveCfg = Debug|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Any CPU.ActiveCfg = Release|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Any CPU.Build.0 = Release|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Mixed Platforms.Build.0 = Release|Any CPU - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|x86.ActiveCfg = Release|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Any CPU.Build.0 = Debug|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|x86.ActiveCfg = Debug|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Any CPU.ActiveCfg = Release|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Any CPU.Build.0 = Release|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Mixed Platforms.Build.0 = Release|Any CPU - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|x86.ActiveCfg = Release|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Any CPU.Build.0 = Debug|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|x86.ActiveCfg = Debug|x86 - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|x86.Build.0 = Debug|x86 - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Any CPU.ActiveCfg = Release|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Any CPU.Build.0 = Release|Any CPU - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Mixed Platforms.ActiveCfg = Release|x86 - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Mixed Platforms.Build.0 = Release|x86 - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|x86.ActiveCfg = Release|x86 - {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|x86.Build.0 = Release|x86 - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Any CPU.Build.0 = Debug|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|x86.ActiveCfg = Debug|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Any CPU.ActiveCfg = Release|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Any CPU.Build.0 = Release|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Mixed Platforms.Build.0 = Release|Any CPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|x86.ActiveCfg = Release|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Any CPU.Build.0 = Debug|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|x86.ActiveCfg = Debug|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Any CPU.ActiveCfg = Release|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Any CPU.Build.0 = Release|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Mixed Platforms.Build.0 = Release|Any CPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|x86.ActiveCfg = Release|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Any CPU.Build.0 = Debug|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|x86.ActiveCfg = Debug|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Any CPU.ActiveCfg = Release|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Any CPU.Build.0 = Release|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Mixed Platforms.Build.0 = Release|Any CPU - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|x86.ActiveCfg = Release|Any CPU - EndGlobalSection - GlobalSection(SolutionProperties) = preSolution - HideSolutionNode = FALSE - EndGlobalSection -EndGlobal + +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio 2013 +VisualStudioVersion = 12.0.40629.0 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.main", "src\apache\main\Avro.main.csproj", "{A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.test", "src\apache\test\Avro.test.csproj", "{911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.codegen", "src\apache\codegen\Avro.codegen.csproj", "{BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.ipc", "src\apache\ipc\Avro.ipc.csproj", "{3B05043A-DC6C-49B6-85BF-9AB055D0B414}" + ProjectSection(ProjectDependencies) = postProject + {AEB22F94-4ECF-4008-B159-389B3F05D54B} = {AEB22F94-4ECF-4008-B159-389B3F05D54B} + EndProjectSection +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.msbuild", "src\apache\msbuild\Avro.msbuild.csproj", "{AEB22F94-4ECF-4008-B159-389B3F05D54B}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Avro.perf", "src\apache\perf\Avro.perf.csproj", "{AC4E1909-2594-4D01-9B2B-B832C07BAFE5}" +EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "globals", "globals", "{C230DE2B-3963-4930-81BF-9421BD297BD9}" + ProjectSection(SolutionItems) = preProject + GlobalAssemblyInfo.cs = GlobalAssemblyInfo.cs + EndProjectSection +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Debug|Mixed Platforms = Debug|Mixed Platforms + Debug|x86 = Debug|x86 + Release|Any CPU = Release|Any CPU + Release|Mixed Platforms = Release|Mixed Platforms + Release|x86 = Release|x86 + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Any CPU.Build.0 = Debug|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Debug|x86.ActiveCfg = Debug|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Any CPU.ActiveCfg = Release|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Any CPU.Build.0 = Release|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|Mixed Platforms.Build.0 = Release|Any CPU + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0}.Release|x86.ActiveCfg = Release|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Any CPU.Build.0 = Debug|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Debug|x86.ActiveCfg = Debug|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Any CPU.ActiveCfg = Release|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Any CPU.Build.0 = Release|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|Mixed Platforms.Build.0 = Release|Any CPU + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA}.Release|x86.ActiveCfg = Release|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Any CPU.Build.0 = Debug|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|x86.ActiveCfg = Debug|x86 + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Debug|x86.Build.0 = Debug|x86 + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Any CPU.ActiveCfg = Release|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Any CPU.Build.0 = Release|Any CPU + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Mixed Platforms.ActiveCfg = Release|x86 + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|Mixed Platforms.Build.0 = Release|x86 + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|x86.ActiveCfg = Release|x86 + {BF0D313C-1AA3-4900-B277-B0F5F9DDCDA8}.Release|x86.Build.0 = Release|x86 + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Any CPU.Build.0 = Debug|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Debug|x86.ActiveCfg = Debug|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Any CPU.ActiveCfg = Release|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Any CPU.Build.0 = Release|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|Mixed Platforms.Build.0 = Release|Any CPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414}.Release|x86.ActiveCfg = Release|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Any CPU.Build.0 = Debug|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Debug|x86.ActiveCfg = Debug|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Any CPU.ActiveCfg = Release|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Any CPU.Build.0 = Release|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|Mixed Platforms.Build.0 = Release|Any CPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B}.Release|x86.ActiveCfg = Release|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Any CPU.Build.0 = Debug|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Debug|x86.ActiveCfg = Debug|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Any CPU.ActiveCfg = Release|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Any CPU.Build.0 = Release|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|Mixed Platforms.Build.0 = Release|Any CPU + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5}.Release|x86.ActiveCfg = Release|Any CPU + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection +EndGlobal diff --git a/lang/csharp/GlobalAssemblyInfo.cs b/lang/csharp/GlobalAssemblyInfo.cs new file mode 100644 index 00000000..1ad0fecd --- /dev/null +++ b/lang/csharp/GlobalAssemblyInfo.cs @@ -0,0 +1,24 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +using System.Reflection; + +[assembly: AssemblyCompany("Apache")] +[assembly: AssemblyProduct("Avro")] +[assembly: AssemblyCopyright("Copyright © Apache 2013")] +[assembly: AssemblyVersion("1.7.7.0")] \ No newline at end of file diff --git a/lang/csharp/lib/main/Castle.Core.dll b/lang/csharp/lib/main/Castle.Core.dll deleted file mode 100644 index ccc7d5ff..00000000 Binary files a/lang/csharp/lib/main/Castle.Core.dll and /dev/null differ diff --git a/lang/csharp/lib/main/Newtonsoft.Json.dll b/lang/csharp/lib/main/Newtonsoft.Json.dll deleted file mode 100644 index 4703b890..00000000 Binary files a/lang/csharp/lib/main/Newtonsoft.Json.dll and /dev/null differ diff --git a/lang/csharp/lib/main/log4net.dll b/lang/csharp/lib/main/log4net.dll deleted file mode 100644 index ffc57e11..00000000 Binary files a/lang/csharp/lib/main/log4net.dll and /dev/null differ diff --git a/lang/csharp/lib/test/nunit.framework.dll b/lang/csharp/lib/test/nunit.framework.dll deleted file mode 100644 index 639dbb0d..00000000 Binary files a/lang/csharp/lib/test/nunit.framework.dll and /dev/null differ diff --git a/lang/csharp/src/apache/codegen/Avro.codegen.csproj b/lang/csharp/src/apache/codegen/Avro.codegen.csproj index fe58ab03..aaef5f52 100644 --- a/lang/csharp/src/apache/codegen/Avro.codegen.csproj +++ b/lang/csharp/src/apache/codegen/Avro.codegen.csproj @@ -1,4 +1,4 @@ - + - + \ No newline at end of file diff --git a/lang/csharp/src/apache/codegen/Avro.codegen.nuspec b/lang/csharp/src/apache/codegen/Avro.codegen.nuspec new file mode 100644 index 00000000..5061e3b1 --- /dev/null +++ b/lang/csharp/src/apache/codegen/Avro.codegen.nuspec @@ -0,0 +1,17 @@ + + + + Avro.codegen + $version$ + $title$ + $author$ + $author$ + http://www.apache.org/licenses/#2.0 + https://avro.apache.org/ + https://avro.apache.org/images/avro-logo.png + false + $description$ + Copyright 2015 + apache avro + + \ No newline at end of file diff --git a/lang/csharp/src/apache/codegen/Properties/AssemblyInfo.cs b/lang/csharp/src/apache/codegen/Properties/AssemblyInfo.cs index 0eeb2366..fa25aef8 100644 --- a/lang/csharp/src/apache/codegen/Properties/AssemblyInfo.cs +++ b/lang/csharp/src/apache/codegen/Properties/AssemblyInfo.cs @@ -20,14 +20,7 @@ using System.Runtime.InteropServices; [assembly: AssemblyTitle("Avro.codegen")] -[assembly: AssemblyDescription("")] -[assembly: AssemblyConfiguration("")] -[assembly: AssemblyCompany("Apache")] -[assembly: AssemblyProduct("Avro.codegen")] -[assembly: AssemblyCopyright("Copyright © Apache 2013")] -[assembly: AssemblyTrademark("")] +[assembly: AssemblyDescription("Avro C# code generator")] [assembly: AssemblyCulture("")] [assembly: ComVisible(false)] [assembly: Guid("3C23DD33-DD4F-42B1-B71F-8F9C86929E58")] -[assembly: AssemblyVersion("0.9.0.0")] -[assembly: AssemblyFileVersion("0.9.0.0")] \ No newline at end of file diff --git a/lang/csharp/src/apache/ipc/Avro.ipc.csproj b/lang/csharp/src/apache/ipc/Avro.ipc.csproj index ad0821e3..e57f5a0c 100644 --- a/lang/csharp/src/apache/ipc/Avro.ipc.csproj +++ b/lang/csharp/src/apache/ipc/Avro.ipc.csproj @@ -1,4 +1,4 @@ - + - - - Debug - AnyCPU - {3B05043A-DC6C-49B6-85BF-9AB055D0B414} - Library - Properties - Avro.ipc - Avro.ipc - v4.0 - 512 - - - - true - full - false - ..\..\..\build\ipc\Debug\ - DEBUG;TRACE - prompt - 4 - AllRules.ruleset - - - pdbonly - true - ..\..\..\build\ipc\Release\ - TRACE - prompt - 4 - - - true - - - ..\..\..\Avro.snk - - - - ..\..\..\lib\main\Castle.Core.dll - - - ..\..\..\lib\main\log4net.dll - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - {a0a5ca3c-f58c-4d07-98b0-2c7b62ab20f0} - Avro.main - - - - - +--> + + + Debug + AnyCPU + {3B05043A-DC6C-49B6-85BF-9AB055D0B414} + Library + Properties + Avro.ipc + Avro.ipc + v4.0 + 512 + + + + true + full + false + ..\..\..\build\ipc\Debug\ + DEBUG;TRACE + prompt + 4 + AllRules.ruleset + + + pdbonly + true + ..\..\..\build\ipc\Release\ + TRACE + prompt + 4 + + + true + + + ..\..\..\Avro.snk + + + + ..\..\..\packages\Castle.Core.3.3.3\lib\net40-client\Castle.Core.dll + True + + + ..\..\..\packages\log4net.2.0.3\lib\net40-full\log4net.dll + True + + + + + + + Properties\GlobalAssemblyInfo.cs + + + + + + + + + + + + + + + + + + + + + + + + + + + + {a0a5ca3c-f58c-4d07-98b0-2c7b62ab20f0} + Avro.main + + + + + - + --> + + --> \ No newline at end of file diff --git a/lang/csharp/src/apache/ipc/Properties/AssemblyInfo.cs b/lang/csharp/src/apache/ipc/Properties/AssemblyInfo.cs index 4875ed1a..372053c1 100644 --- a/lang/csharp/src/apache/ipc/Properties/AssemblyInfo.cs +++ b/lang/csharp/src/apache/ipc/Properties/AssemblyInfo.cs @@ -21,13 +21,6 @@ [assembly: AssemblyTitle("Avro.ipc")] [assembly: AssemblyDescription("")] -[assembly: AssemblyConfiguration("")] -[assembly: AssemblyCompany("Apache")] -[assembly: AssemblyProduct("Avro.ipc")] -[assembly: AssemblyCopyright("Copyright © Apache 2013")] -[assembly: AssemblyTrademark("")] [assembly: AssemblyCulture("")] [assembly: ComVisible(false)] [assembly: Guid("CC3EF4B8-52AA-47D9-B170-5BAAA57A38F3")] -[assembly: AssemblyVersion("0.9.0.0")] -[assembly: AssemblyFileVersion("0.9.0.0")] \ No newline at end of file diff --git a/lang/csharp/src/apache/ipc/packages.config b/lang/csharp/src/apache/ipc/packages.config new file mode 100644 index 00000000..785f1757 --- /dev/null +++ b/lang/csharp/src/apache/ipc/packages.config @@ -0,0 +1,5 @@ + + + + + \ No newline at end of file diff --git a/lang/csharp/src/apache/main/Avro.main.csproj b/lang/csharp/src/apache/main/Avro.main.csproj index 7f41e3ff..b0d03556 100644 --- a/lang/csharp/src/apache/main/Avro.main.csproj +++ b/lang/csharp/src/apache/main/Avro.main.csproj @@ -1,4 +1,4 @@ - + - - - Debug - AnyCPU - 8.0.30703 - 2.0 - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} - Library - Properties - Avro - Avro - v3.5 - 512 - - - 3.5 - - publish\ - true - Disk - false - Foreground - 7 - Days - false - false - true - 0 - 1.0.0.%2a - false - false - true - - - - true - full - false - ..\..\..\build\main\Debug\ - DEBUG;TRACE - prompt - 4 - Off - - - none - true - ..\..\..\build\main\Release\ - - - prompt - 4 - Off - - - true - - - ..\..\..\Avro.snk - - - - ..\..\..\lib\main\log4net.dll - - - ..\..\..\lib\main\Newtonsoft.Json.dll - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - False - Microsoft .NET Framework 4 %28x86 and x64%29 - true - - - False - .NET Framework 3.5 SP1 - false - - - False - Windows Installer 3.1 - true - - - +--> + + + Debug + AnyCPU + 8.0.30703 + 2.0 + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} + Library + Properties + Avro + Avro + v3.5 + 512 + + + 3.5 + + publish\ + true + Disk + false + Foreground + 7 + Days + false + false + true + 0 + 1.0.0.%2a + false + false + true + + + + true + full + false + ..\..\..\build\main\Debug\ + DEBUG;TRACE + prompt + 4 + Off + + + none + true + ..\..\..\build\main\Release\ + + + prompt + 4 + Off + + + true + + + ..\..\..\Avro.snk + + + + ..\..\..\packages\log4net.2.0.3\lib\net35-full\log4net.dll + True + + + ..\..\..\packages\Newtonsoft.Json.7.0.1\lib\net35\Newtonsoft.Json.dll + True + + + + + + Properties\GlobalAssemblyInfo.cs + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + False + Microsoft .NET Framework 4 %28x86 and x64%29 + true + + + False + .NET Framework 3.5 SP1 + false + + + False + Windows Installer 3.1 + true + + + + + + + --> \ No newline at end of file diff --git a/lang/csharp/src/apache/main/Avro.main.nuspec b/lang/csharp/src/apache/main/Avro.main.nuspec new file mode 100644 index 00000000..0187f276 --- /dev/null +++ b/lang/csharp/src/apache/main/Avro.main.nuspec @@ -0,0 +1,17 @@ + + + + Avro.main + $version$ + $title$ + $author$ + $author$ + http://www.apache.org/licenses/#2.0 + https://avro.apache.org/ + https://avro.apache.org/images/avro-logo.png + false + $description$ + Copyright 2015 + apache avro + + \ No newline at end of file diff --git a/lang/csharp/src/apache/main/Properties/AssemblyInfo.cs b/lang/csharp/src/apache/main/Properties/AssemblyInfo.cs index 1ef0b2ef..e16f7b54 100644 --- a/lang/csharp/src/apache/main/Properties/AssemblyInfo.cs +++ b/lang/csharp/src/apache/main/Properties/AssemblyInfo.cs @@ -20,14 +20,8 @@ using System.Runtime.InteropServices; [assembly: AssemblyTitle("Avro")] -[assembly: AssemblyDescription("")] -[assembly: AssemblyConfiguration("")] -[assembly: AssemblyCompany("Apache")] -[assembly: AssemblyProduct("Avro")] -[assembly: AssemblyCopyright("Copyright © Apache 2013")] -[assembly: AssemblyTrademark("")] +[assembly: AssemblyDescription("Avro C# library")] [assembly: AssemblyCulture("")] [assembly: ComVisible(false)] [assembly: Guid("152D7B83-9A97-45F3-B4B3-A367AFC090C4")] -[assembly: AssemblyVersion("0.9.0.0")] -[assembly: AssemblyFileVersion("0.9.0.0")] \ No newline at end of file + diff --git a/lang/csharp/src/apache/main/Schema/Property.cs b/lang/csharp/src/apache/main/Schema/Property.cs index 3901623e..b3ea4d32 100644 --- a/lang/csharp/src/apache/main/Schema/Property.cs +++ b/lang/csharp/src/apache/main/Schema/Property.cs @@ -78,7 +78,7 @@ public void WriteJson(JsonTextWriter writer) if (ReservedProps.Contains(kp.Key)) continue; writer.WritePropertyName(kp.Key); - writer.WriteRawValue(kp.Value); + writer.WriteValue(kp.Value); } } diff --git a/lang/csharp/src/apache/main/packages.config b/lang/csharp/src/apache/main/packages.config new file mode 100644 index 00000000..483c203a --- /dev/null +++ b/lang/csharp/src/apache/main/packages.config @@ -0,0 +1,5 @@ + + + + + \ No newline at end of file diff --git a/lang/csharp/src/apache/make.nuget.cmd b/lang/csharp/src/apache/make.nuget.cmd new file mode 100644 index 00000000..2fb66f19 --- /dev/null +++ b/lang/csharp/src/apache/make.nuget.cmd @@ -0,0 +1,8 @@ +cd main +nuget pack -Build -Symbols -Properties Configuration=Release + +cd ..\codegen +nuget pack -Build -Symbols -Properties Configuration=Release + +cd ..\msbuild +nuget pack -Build -Symbols -Properties Configuration=Release diff --git a/lang/csharp/src/apache/msbuild/Avro.msbuild.csproj b/lang/csharp/src/apache/msbuild/Avro.msbuild.csproj index f1dd6f8f..d372e400 100644 --- a/lang/csharp/src/apache/msbuild/Avro.msbuild.csproj +++ b/lang/csharp/src/apache/msbuild/Avro.msbuild.csproj @@ -1,4 +1,4 @@ - + - - - - Debug - AnyCPU - {AEB22F94-4ECF-4008-B159-389B3F05D54B} - Library - Properties - Avro.msbuild - Avro.msbuild - v3.5 - 512 - - - - true - full - false - ..\..\..\build\msbuild\Debug\ - DEBUG;TRACE - prompt - 4 - - - pdbonly - true - ..\..\..\build\msbuild\Release\ - TRACE - prompt - 4 - - - - - - - - - - - - - - - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} - Avro.main - - - +--> + + + + Debug + AnyCPU + {AEB22F94-4ECF-4008-B159-389B3F05D54B} + Library + Properties + Avro.msbuild + Avro.msbuild + v3.5 + 512 + + + + true + full + false + ..\..\..\build\msbuild\Debug\ + DEBUG;TRACE + prompt + 4 + + + pdbonly + true + ..\..\..\build\msbuild\Release\ + TRACE + prompt + 4 + + + + + + + + + + + Properties\GlobalAssemblyInfo.cs + + + + + + + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} + Avro.main + + + + --> \ No newline at end of file diff --git a/lang/csharp/src/apache/msbuild/Avro.msbuild.nuspec b/lang/csharp/src/apache/msbuild/Avro.msbuild.nuspec new file mode 100644 index 00000000..87c378d5 --- /dev/null +++ b/lang/csharp/src/apache/msbuild/Avro.msbuild.nuspec @@ -0,0 +1,18 @@ + + + + $id$ + $version$ + $title$ + $author$ + $author$ + http://www.apache.org/licenses/#2.0 + https://avro.apache.org/ + https://avro.apache.org/images/avro-logo.png + false + $description$ + Copyright 2015 + apache avro + true + + \ No newline at end of file diff --git a/lang/csharp/src/apache/msbuild/Properties/AssemblyInfo.cs b/lang/csharp/src/apache/msbuild/Properties/AssemblyInfo.cs index e46675b1..41d965c2 100644 --- a/lang/csharp/src/apache/msbuild/Properties/AssemblyInfo.cs +++ b/lang/csharp/src/apache/msbuild/Properties/AssemblyInfo.cs @@ -20,14 +20,7 @@ using System.Runtime.InteropServices; [assembly: AssemblyTitle("Avro.msbuild")] -[assembly: AssemblyDescription("")] -[assembly: AssemblyConfiguration("")] -[assembly: AssemblyCompany("Apache")] -[assembly: AssemblyProduct("Avro.msbuild")] -[assembly: AssemblyCopyright("Copyright © Apache 2013")] -[assembly: AssemblyTrademark("")] +[assembly: AssemblyDescription("Avro msbuild codegen task")] [assembly: AssemblyCulture("")] [assembly: ComVisible(false)] [assembly: Guid("A7982FA5-F369-457F-894E-4F90262E4638")] -[assembly: AssemblyVersion("0.9.0.0")] -[assembly: AssemblyFileVersion("0.9.0.0")] \ No newline at end of file diff --git a/lang/csharp/src/apache/perf/Avro.perf.csproj b/lang/csharp/src/apache/perf/Avro.perf.csproj index 327b6144..9383b286 100644 --- a/lang/csharp/src/apache/perf/Avro.perf.csproj +++ b/lang/csharp/src/apache/perf/Avro.perf.csproj @@ -1,4 +1,4 @@ - + - - - Debug - AnyCPU - 8.0.30703 - 2.0 - {AC4E1909-2594-4D01-9B2B-B832C07BAFE5} - Exe - Properties - Avro.perf - Avro.perf - v4.0 - 512 - - - 3.5 - - publish\ - true - Disk - false - Foreground - 7 - Days - false - false - true - 0 - 1.0.0.%2a - false - false - true - - - true - full - false - ..\..\..\build\perf\Debug\ - DEBUG;TRACE - prompt - 4 - Off - - - none - true - ..\..\..\build\perf\Release\ - - - prompt - 4 - Off - - - - - - - False - ..\..\..\..\..\lang\csharp\lib\main\Castle.Core.dll - - - False - ..\..\..\lib\test\nunit.framework.dll - - - - - - - - - - - {3b05043a-dc6c-49b6-85bf-9ab055d0b414} - Avro.ipc - - - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} - Avro.main - - - - - False - Microsoft .NET Framework 4 %28x86 and x64%29 - true - - - False - .NET Framework 3.5 SP1 - false - - - False - Windows Installer 3.1 - true - - - - - - - - - - - - - Code - - - - - - +--> + + + Debug + AnyCPU + 8.0.30703 + 2.0 + {AC4E1909-2594-4D01-9B2B-B832C07BAFE5} + Exe + Properties + Avro.perf + Avro.perf + v4.0 + 512 + + + 3.5 + + publish\ + true + Disk + false + Foreground + 7 + Days + false + false + true + 0 + 1.0.0.%2a + false + false + true + + + true + full + false + ..\..\..\build\perf\Debug\ + DEBUG;TRACE + prompt + 4 + Off + + + none + true + ..\..\..\build\perf\Release\ + + + prompt + 4 + Off + + + + + + + + + + + {3b05043a-dc6c-49b6-85bf-9ab055d0b414} + Avro.ipc + + + {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} + Avro.main + + + + + False + Microsoft .NET Framework 4 %28x86 and x64%29 + true + + + False + .NET Framework 3.5 SP1 + false + + + False + Windows Installer 3.1 + true + + + + + + + + + + + + + Code + + + + + + + --> \ No newline at end of file diff --git a/lang/csharp/src/apache/test/Avro.test.csproj b/lang/csharp/src/apache/test/Avro.test.csproj index 34babf7d..cb058b6d 100644 --- a/lang/csharp/src/apache/test/Avro.test.csproj +++ b/lang/csharp/src/apache/test/Avro.test.csproj @@ -1,4 +1,4 @@ - + - - - Debug - AnyCPU - 8.0.30703 - 2.0 - {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA} - Library - Properties - Avro.test - Avro.test - v4.0 - 512 - - - 3.5 - - publish\ - true - Disk - false - Foreground - 7 - Days - false - false - true - 0 - 1.0.0.%2a - false - false - true - - - true - full - false - ..\..\..\build\test\Debug\ - DEBUG;TRACE - prompt - 4 - Off - - - none - true - ..\..\..\build\test\Release\ - - - prompt - 4 - Off - - - - False - ..\..\..\..\..\lang\csharp\lib\main\Castle.Core.dll - - - False - ..\..\..\lib\test\nunit.framework.dll - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - {3b05043a-dc6c-49b6-85bf-9ab055d0b414} - Avro.ipc - - - {A0A5CA3C-F58C-4D07-98B0-2C7B62AB20F0} - Avro.main - - - - - False - Microsoft .NET Framework 4 %28x86 and x64%29 - true - - - False - .NET Framework 3.5 SP1 - false - - - False - Windows Installer 3.1 - true - - - - - Ipc\mail.avpr - - - - +--> + + + Debug + AnyCPU + 8.0.30703 + 2.0 + {911D56AB-587B-4E5F-B5EA-D47D8A46F1FA} + Library + Properties + Avro.test + Avro.test + v4.0 + 512 + + + 3.5 + + publish\ + true + Disk + false + Foreground + 7 + Days + false + false + true + 0 + 1.0.0.%2a + false + false + true + + + true + full + false + ..\..\..\build\test\Debug\ + DEBUG;TRACE + prompt + 4 + Off + + + none + true + ..\..\..\build\test\Release\ + + + prompt + 4 + Off + + + + ..\..\..\packages\Castle.Core.3.3.3\lib\net40-client\Castle.Core.dll + True + + + ..\..\..\packages\NUnit.2.6.4\lib\nunit.framework.dll + True + + + + + + + + + + + Properties\GlobalAssemblyInfo.cs + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + False + Microsoft .NET Framework 4 %28x86 and x64%29 + true + + + False + .NET Framework 3.5 SP1 + false + + + False + Windows Installer 3.1 + true + + + + + Ipc\mail.avpr + + + + + {3b05043a-dc6c-49b6-85bf-9ab055d0b414} + Avro.ipc + + + {a0a5ca3c-f58c-4d07-98b0-2c7b62ab20f0} + Avro.main + + + + + + + --> \ No newline at end of file diff --git a/lang/csharp/src/apache/test/Properties/AssemblyInfo.cs b/lang/csharp/src/apache/test/Properties/AssemblyInfo.cs index 8821f64d..0adcc68b 100644 --- a/lang/csharp/src/apache/test/Properties/AssemblyInfo.cs +++ b/lang/csharp/src/apache/test/Properties/AssemblyInfo.cs @@ -21,13 +21,6 @@ [assembly: AssemblyTitle("Avro.test")] [assembly: AssemblyDescription("")] -[assembly: AssemblyConfiguration("")] -[assembly: AssemblyCompany("Apache")] -[assembly: AssemblyProduct("Avro.test")] -[assembly: AssemblyCopyright("Copyright © Apache 2013")] -[assembly: AssemblyTrademark("")] [assembly: AssemblyCulture("")] [assembly: ComVisible(false)] [assembly: Guid("442785CE-3633-4A04-A103-434104F63D55")] -[assembly: AssemblyVersion("0.9.0.0")] -[assembly: AssemblyFileVersion("0.9.0.0")] \ No newline at end of file diff --git a/lang/csharp/src/apache/test/packages.config b/lang/csharp/src/apache/test/packages.config new file mode 100644 index 00000000..55272244 --- /dev/null +++ b/lang/csharp/src/apache/test/packages.config @@ -0,0 +1,5 @@ + + + + + \ No newline at end of file diff --git a/lang/java/archetypes/avro-service-archetype/pom.xml b/lang/java/archetypes/avro-service-archetype/pom.xml index 3aaca743..361a9092 100644 --- a/lang/java/archetypes/avro-service-archetype/pom.xml +++ b/lang/java/archetypes/avro-service-archetype/pom.xml @@ -23,7 +23,7 @@ avro-archetypes-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../pom.xml diff --git a/lang/java/archetypes/pom.xml b/lang/java/archetypes/pom.xml index 9bce9ebd..5733d157 100644 --- a/lang/java/archetypes/pom.xml +++ b/lang/java/archetypes/pom.xml @@ -22,7 +22,7 @@ org.apache.avro avro-parent - 1.7.7-SNAPSHOT + 1.7.7 ../pom.xml diff --git a/lang/java/avro/pom.xml b/lang/java/avro/pom.xml index 7d982e16..0e672117 100644 --- a/lang/java/avro/pom.xml +++ b/lang/java/avro/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/avro/src/main/java/org/apache/avro/Schema.java b/lang/java/avro/src/main/java/org/apache/avro/Schema.java index b9e3ba4e..321b0bea 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/Schema.java +++ b/lang/java/avro/src/main/java/org/apache/avro/Schema.java @@ -432,15 +432,15 @@ public Name(String name, String space) { return; } int lastDot = name.lastIndexOf('.'); - if ("".equals(space)) - space = null; if (lastDot < 0) { // unqualified name - this.space = space; // use default space this.name = validateName(name); } else { // qualified name - this.space = name.substring(0, lastDot); // get space from name + space = name.substring(0, lastDot); // get space from name this.name = validateName(name.substring(lastDot+1, name.length())); } + if ("".equals(space)) + space = null; + this.space = space; this.full = (this.space == null) ? this.name : this.space+"."+this.name; } public boolean equals(Object o) { diff --git a/lang/java/avro/src/main/java/org/apache/avro/SchemaValidationStrategy.java b/lang/java/avro/src/main/java/org/apache/avro/SchemaValidationStrategy.java index 6c261f77..dc1c9cc0 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/SchemaValidationStrategy.java +++ b/lang/java/avro/src/main/java/org/apache/avro/SchemaValidationStrategy.java @@ -1,3 +1,21 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + package org.apache.avro; /** diff --git a/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java b/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java index 1be8c4c3..4c361921 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java +++ b/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java @@ -502,9 +502,6 @@ private void writeEscapedString(String string, StringBuilder builder) { case '\t': builder.append("\\t"); break; - case '/': - builder.append("\\/"); - break; default: // Reference: http://www.unicode.org/versions/Unicode5.1.0/ if((ch>='\u0000' && ch<='\u001F') || (ch>='\u007F' && ch<='\u009F') || (ch>='\u2000' && ch<='\u20FF')){ diff --git a/lang/java/avro/src/main/java/org/apache/avro/io/ResolvingDecoder.java b/lang/java/avro/src/main/java/org/apache/avro/io/ResolvingDecoder.java index 62947be8..2d7eba2a 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/io/ResolvingDecoder.java +++ b/lang/java/avro/src/main/java/org/apache/avro/io/ResolvingDecoder.java @@ -18,11 +18,14 @@ package org.apache.avro.io; import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.charset.Charset; import org.apache.avro.AvroTypeException; import org.apache.avro.Schema; import org.apache.avro.io.parsing.ResolvingGrammarGenerator; import org.apache.avro.io.parsing.Symbol; +import org.apache.avro.util.Utf8; /** * {@link Decoder} that performs type-resolution between the reader's and @@ -188,6 +191,64 @@ public double readDouble() throws IOException { } } + @Override + public Utf8 readString(Utf8 old) throws IOException { + Symbol actual = parser.advance(Symbol.STRING); + if (actual == Symbol.BYTES) { + return new Utf8(in.readBytes(null).array()); + } else { + assert actual == Symbol.STRING; + return in.readString(old); + } + } + + private static final Charset UTF8 = Charset.forName("UTF-8"); + + @Override + public String readString() throws IOException { + Symbol actual = parser.advance(Symbol.STRING); + if (actual == Symbol.BYTES) { + return new String(in.readBytes(null).array(), UTF8); + } else { + assert actual == Symbol.STRING; + return in.readString(); + } + } + + @Override + public void skipString() throws IOException { + Symbol actual = parser.advance(Symbol.STRING); + if (actual == Symbol.BYTES) { + in.skipBytes(); + } else { + assert actual == Symbol.STRING; + in.skipString(); + } + } + + @Override + public ByteBuffer readBytes(ByteBuffer old) throws IOException { + Symbol actual = parser.advance(Symbol.BYTES); + if (actual == Symbol.STRING) { + Utf8 s = in.readString(null); + return ByteBuffer.wrap(s.getBytes(), 0, s.getByteLength()); + } else { + assert actual == Symbol.BYTES; + return in.readBytes(old); + } + } + + @Override + public void skipBytes() throws IOException { + Symbol actual = parser.advance(Symbol.BYTES); + if (actual == Symbol.STRING) { + in.skipString(); + } else { + assert actual == Symbol.BYTES; + in.skipBytes(); + } + } + @Override public int readEnum() throws IOException { parser.advance(Symbol.ENUM); diff --git a/lang/java/avro/src/main/java/org/apache/avro/io/parsing/ResolvingGrammarGenerator.java b/lang/java/avro/src/main/java/org/apache/avro/io/parsing/ResolvingGrammarGenerator.java index a7054099..a4ae238e 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/io/parsing/ResolvingGrammarGenerator.java +++ b/lang/java/avro/src/main/java/org/apache/avro/io/parsing/ResolvingGrammarGenerator.java @@ -151,6 +151,20 @@ public Symbol generate(Schema writer, Schema reader, } break; + case BYTES: + switch (writerType) { + case STRING: + return Symbol.resolve(super.generate(writer, seen), Symbol.BYTES); + } + break; + + case STRING: + switch (writerType) { + case BYTES: + return Symbol.resolve(super.generate(writer, seen), Symbol.STRING); + } + break; + case UNION: int j = bestBranch(reader, writer); if (j >= 0) { @@ -161,8 +175,6 @@ public Symbol generate(Schema writer, Schema reader, case NULL: case BOOLEAN: case INT: - case STRING: - case BYTES: case ENUM: case ARRAY: case MAP: @@ -449,6 +461,18 @@ private static int bestBranch(Schema r, Schema w) { return j; } break; + case STRING: + switch (b.getType()) { + case BYTES: + return j; + } + break; + case BYTES: + switch (b.getType()) { + case STRING: + return j; + } + break; } j++; } diff --git a/lang/java/avro/src/main/java/org/apache/avro/reflect/FieldAccessReflect.java b/lang/java/avro/src/main/java/org/apache/avro/reflect/FieldAccessReflect.java index 5b79f2c8..680139ac 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/reflect/FieldAccessReflect.java +++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/FieldAccessReflect.java @@ -17,6 +17,10 @@ */ package org.apache.avro.reflect; +import org.apache.avro.AvroRuntimeException; +import org.apache.avro.io.Decoder; +import org.apache.avro.io.Encoder; + import java.io.IOException; import java.lang.reflect.Field; @@ -24,11 +28,18 @@ class FieldAccessReflect extends FieldAccess { @Override protected FieldAccessor getAccessor(Field field) { + AvroEncode enc = field.getAnnotation(AvroEncode.class); + if (enc != null) + try { + return new ReflectionBasesAccessorCustomEncoded(field, enc.using().newInstance()); + } catch (Exception e) { + throw new AvroRuntimeException("Could not instantiate custom Encoding"); + } return new ReflectionBasedAccessor(field); } - private final class ReflectionBasedAccessor extends FieldAccessor { - private final Field field; + private class ReflectionBasedAccessor extends FieldAccessor { + protected final Field field; private boolean isStringable; private boolean isCustomEncoded; @@ -69,6 +80,42 @@ protected boolean isStringable() { protected boolean isCustomEncoded() { return isCustomEncoded; } + } + + private final class ReflectionBasesAccessorCustomEncoded extends ReflectionBasedAccessor { + + private CustomEncoding encoding; + + public ReflectionBasesAccessorCustomEncoded(Field f, CustomEncoding encoding) { + super(f); + this.encoding = encoding; + } + + @Override + protected void read(Object object, Decoder in) throws IOException { + try { + field.set(object, encoding.read(in)); + } catch (IllegalAccessException e) { + throw new AvroRuntimeException(e); +} + } + @Override + protected void write(Object object, Encoder out) throws IOException { + try { + encoding.write(field.get(object), out); + } catch (IllegalAccessException e) { + throw new AvroRuntimeException(e); + } + } + + protected boolean isCustomEncoded() { + return true; + } + + @Override + protected boolean supportsIO() { + return true; + } } } diff --git a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java index 799e63eb..113752c8 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java +++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java @@ -206,7 +206,7 @@ public boolean validate(Schema schema, Object datum) { } } - private static final ConcurrentHashMap, ClassAccessorData> + static final ConcurrentHashMap, ClassAccessorData> ACCESSOR_CACHE = new ConcurrentHashMap, ClassAccessorData>(); private static class ClassAccessorData { diff --git a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectionUtil.java b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectionUtil.java index 47b4f296..1457cdbf 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectionUtil.java +++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectionUtil.java @@ -33,8 +33,11 @@ class ReflectionUtil { private ReflectionUtil() { } - private static final FieldAccess FIELD_ACCESS; + private static FieldAccess fieldAccess; static { + resetFieldAccess(); + } + static void resetFieldAccess() { // load only one implementation of FieldAccess // so it is monomorphic and the JIT can inline FieldAccess access = null; @@ -60,7 +63,7 @@ private ReflectionUtil() { "Unable to load a functional FieldAccess class!"); } } - FIELD_ACCESS = access; + fieldAccess = access; } private static T load(String name, Class type) throws Exception { @@ -69,7 +72,7 @@ private static T load(String name, Class type) throws Exception { } public static FieldAccess getFieldAccess() { - return FIELD_ACCESS; + return fieldAccess; } private static boolean validate(FieldAccess access) throws Exception { diff --git a/lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericData.java b/lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericData.java index 4e2f1839..b69322d9 100644 --- a/lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericData.java +++ b/lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericData.java @@ -318,6 +318,11 @@ public void testToStringIsJson() throws JsonParseException, IOException { mapper.readTree(parser); } + @Test public void testToStringDoesNotEscapeForwardSlash() throws Exception { + GenericData data = GenericData.get(); + assertEquals("\"/\"", data.toString("/")); + } + @Test public void testToStringNanInfinity() throws Exception { GenericData data = GenericData.get(); assertEquals("\"Infinity\"",data.toString(Float.POSITIVE_INFINITY)); diff --git a/lang/java/avro/src/test/java/org/apache/avro/io/TestResolvingIO.java b/lang/java/avro/src/test/java/org/apache/avro/io/TestResolvingIO.java index 1369dd88..d5f1b06d 100644 --- a/lang/java/avro/src/test/java/org/apache/avro/io/TestResolvingIO.java +++ b/lang/java/avro/src/test/java/org/apache/avro/io/TestResolvingIO.java @@ -184,12 +184,14 @@ private static Object[][] testSchemas() { + "{\"name\":\"f0\", \"type\":\"boolean\"}," + "{\"name\":\"f1\", \"type\":\"int\"}," + "{\"name\":\"f2\", \"type\":\"float\"}," - + "{\"name\":\"f3\", \"type\":\"string\"}]}", "BIFS", + + "{\"name\":\"f3\", \"type\":\"bytes\"}," + + "{\"name\":\"f4\", \"type\":\"string\"}]}", "BIFbS", "{\"type\":\"record\",\"name\":\"r\",\"fields\":[" + "{\"name\":\"f0\", \"type\":\"boolean\"}," + "{\"name\":\"f1\", \"type\":\"long\"}," + "{\"name\":\"f2\", \"type\":\"double\"}," - + "{\"name\":\"f3\", \"type\":\"string\"}]}", "BLDS" }, + + "{\"name\":\"f3\", \"type\":\"string\"}," + + "{\"name\":\"f4\", \"type\":\"bytes\"}]}", "BLDSb" }, { "[\"int\"]", "U0I", "[\"long\"]", "U0L" }, diff --git a/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java b/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java index 183c6ac6..6c29ccce 100644 --- a/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java +++ b/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java @@ -719,6 +719,24 @@ public void testRecordWithNullIO() throws IOException { assertEquals(b, decoded); } + @Test public void testDisableUnsafe() throws Exception { + String saved = System.getProperty("avro.disable.unsafe"); + try { + System.setProperty("avro.disable.unsafe", "true"); + ReflectData.ACCESSOR_CACHE.clear(); + ReflectionUtil.resetFieldAccess(); + testMultipleAnnotations(); + testRecordWithNullIO(); + } finally { + if (saved == null) + System.clearProperty("avro.disable.unsafe"); + else + System.setProperty("avro.disable.unsafe", saved); + ReflectData.ACCESSOR_CACHE.clear(); + ReflectionUtil.resetFieldAccess(); + } + } + public static class SampleRecord { public int x = 1; private int y = 2; diff --git a/lang/java/compiler/pom.xml b/lang/java/compiler/pom.xml index 3f47f252..c834ff67 100644 --- a/lang/java/compiler/pom.xml +++ b/lang/java/compiler/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/ipc/pom.xml b/lang/java/ipc/pom.xml index 35fdeafd..17e9e0ef 100644 --- a/lang/java/ipc/pom.xml +++ b/lang/java/ipc/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/ipc/src/test/java/org/apache/avro/TestSchema.java b/lang/java/ipc/src/test/java/org/apache/avro/TestSchema.java index 58c1074e..bf9b84ee 100644 --- a/lang/java/ipc/src/test/java/org/apache/avro/TestSchema.java +++ b/lang/java/ipc/src/test/java/org/apache/avro/TestSchema.java @@ -525,6 +525,17 @@ public void testNestedNonNullNamespace2() throws Exception { assertEquals(nullOuter, Schema.parse(nullOuter.toString())); } + @Test + public void testNullNamespaceAlias() throws Exception { + Schema s = + Schema.parse("{\"type\":\"record\",\"name\":\"Z\",\"fields\":[]}"); + Schema t = + Schema.parse("{\"type\":\"record\",\"name\":\"x.Y\",\"aliases\":[\".Z\"]," + +"\"fields\":[]}"); + Schema u = Schema.applyAliases(s, t); + assertEquals("x.Y", u.getFullName()); + } + @Test public void testNullPointer() throws Exception { String recordJson = "{\"type\":\"record\", \"name\":\"Test\", \"fields\":" diff --git a/lang/java/mapred/pom.xml b/lang/java/mapred/pom.xml index fa198bc7..6424271b 100644 --- a/lang/java/mapred/pom.xml +++ b/lang/java/mapred/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/maven-plugin/pom.xml b/lang/java/maven-plugin/pom.xml index 861d0044..d141ea6f 100644 --- a/lang/java/maven-plugin/pom.xml +++ b/lang/java/maven-plugin/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/pom.xml b/lang/java/pom.xml index fcad2804..3bc937f7 100644 --- a/lang/java/pom.xml +++ b/lang/java/pom.xml @@ -22,7 +22,7 @@ org.apache.avro avro-toplevel - 1.7.7-SNAPSHOT + 1.7.7 ../../ diff --git a/lang/java/protobuf/pom.xml b/lang/java/protobuf/pom.xml index 5b3ca623..e950c05c 100644 --- a/lang/java/protobuf/pom.xml +++ b/lang/java/protobuf/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/thrift/pom.xml b/lang/java/thrift/pom.xml index a2b100de..97275d83 100644 --- a/lang/java/thrift/pom.xml +++ b/lang/java/thrift/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/tools/pom.xml b/lang/java/tools/pom.xml index 6e0fe833..77ee4e81 100644 --- a/lang/java/tools/pom.xml +++ b/lang/java/tools/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/trevni/avro/pom.xml b/lang/java/trevni/avro/pom.xml index 1eb9fdf3..622f442b 100644 --- a/lang/java/trevni/avro/pom.xml +++ b/lang/java/trevni/avro/pom.xml @@ -22,7 +22,7 @@ trevni-java org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/trevni/core/pom.xml b/lang/java/trevni/core/pom.xml index f0ea413a..73c2e83c 100644 --- a/lang/java/trevni/core/pom.xml +++ b/lang/java/trevni/core/pom.xml @@ -22,7 +22,7 @@ trevni-java org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/java/trevni/doc/pom.xml b/lang/java/trevni/doc/pom.xml index 079d27e6..08c36716 100644 --- a/lang/java/trevni/doc/pom.xml +++ b/lang/java/trevni/doc/pom.xml @@ -22,13 +22,13 @@ trevni-java org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 .. org.apache.avro trevni-doc - 1.7.7-SNAPSHOT + 1.7.7 pom Trevni Specification diff --git a/lang/java/trevni/pom.xml b/lang/java/trevni/pom.xml index 8be5d6db..ee99b023 100644 --- a/lang/java/trevni/pom.xml +++ b/lang/java/trevni/pom.xml @@ -23,7 +23,7 @@ avro-parent org.apache.avro - 1.7.7-SNAPSHOT + 1.7.7 ../ diff --git a/lang/py3/avro/tests/test_script.py b/lang/py3/avro/tests/test_script.py index 544c4809..8fc72715 100644 --- a/lang/py3/avro/tests/test_script.py +++ b/lang/py3/avro/tests/test_script.py @@ -105,14 +105,11 @@ def RunScript(*args, stdin=None): return out -# The trailing spaces are expected when pretty-printing JSON with json.dumps(): -_JSON_PRETTY = '\n'.join([ - '{', - ' "first": "daffy", ', - ' "last": "duck", ', - ' "type": "duck"', - '}', -]) +_TEST_JSON_VALUE = { + 'first': 'daffy', + 'last': 'duck', + 'type': 'duck', +} # ------------------------------------------------------------------------------ @@ -193,12 +190,12 @@ def testHelp(self): def testJsonPretty(self): out = self._RunCat('--format', 'json-pretty', '-n', '1') self.assertEqual( - out.strip(), - _JSON_PRETTY.strip(), + json.loads(out), + _TEST_JSON_VALUE, 'Output mismatch\n' 'Expect: %r\n' 'Actual: %r' - % (_JSON_PRETTY.strip(), out.strip())) + % (_TEST_JSON_VALUE, out)) def testVersion(self): out = RunScript('cat', '--version').decode('utf-8') diff --git a/lang/py3/setup.py b/lang/py3/setup.py index 3488034b..426ad1dd 100644 --- a/lang/py3/setup.py +++ b/lang/py3/setup.py @@ -25,20 +25,48 @@ from setuptools import setup -def Main(): - assert (sys.version_info[0] >= 3), \ - ('Python version >= 3 required, got %r' % sys.version_info) +VERSION_FILE_NAME = 'VERSION.txt' + + +def RunsFromSourceDist(): + """Tests whether setup.py is invoked from a source distribution. + + Returns: + True if setup.py runs from a source distribution. + False otherwise, ie. if setup.py runs from the SVN trunk. + """ + setup_file_path = os.path.abspath(__file__) + # If a file PKG-INFO exists as a sibling of setup.py, + # assume we are running as source distribution: + pkg_info_file_path = \ + os.path.join(os.path.dirname(setup_file_path), 'PKG-INFO') + return os.path.exists(pkg_info_file_path) + +def SetupSources(): + """Prepares the source directory. + + Runs when setup.py is invoked from the Avro SVN/Git source. + """ + # Avro lang/py3/ source directory: py3_dir = os.path.dirname(os.path.abspath(__file__)) + + # Avro top-level source directory: root_dir = os.path.dirname(os.path.dirname(py3_dir)) + # Copy README.txt from Avro top-level directory: + shutil.copy( + src=os.path.join(root_dir, 'README.txt'), + dst=os.path.join(py3_dir, 'README.txt'), + ) + # Read and copy Avro version: - version_file_path = os.path.join(root_dir, 'share', 'VERSION.txt') + version_file_path = os.path.join(root_dir, 'share', VERSION_FILE_NAME) with open(version_file_path, 'r') as f: avro_version = f.read().strip() shutil.copy( src=version_file_path, - dst=os.path.join(py3_dir, 'avro', 'VERSION.txt'), + dst=os.path.join(py3_dir, 'avro', VERSION_FILE_NAME), ) # Copy necessary avsc files: @@ -71,32 +99,51 @@ def Main(): mode=0o777, ) + +def ReadVersion(): + """Returns: the content of the Avro version file.""" + setup_file_path = os.path.abspath(__file__) + install_dir = os.path.dirname(setup_file_path) + version_file_path = os.path.join(install_dir, 'avro', VERSION_FILE_NAME) + with open(version_file_path, 'rt') as f: + avro_version = f.read().strip() + return avro_version + + +def Main(): + assert (sys.version_info[0] >= 3), \ + ('Python version >= 3 required, got %r' % sys.version_info) + + if not RunsFromSourceDist(): + SetupSources() + + avro_version = ReadVersion() + setup( - name = 'avro-python3', - version = avro_version, - packages = ['avro'], - package_dir = {'avro': 'avro'}, - scripts = ['scripts/avro'], - - include_package_data=True, - package_data = { - 'avro': [ - 'HandshakeRequest.avsc', - 'HandshakeResponse.avsc', - 'VERSION.txt', - ], - }, - - test_suite='avro.tests.run_tests', - tests_require=[], - - # metadata for upload to PyPI - author = 'Apache Avro', - author_email = 'avro-dev@hadoop.apache.org', - description = 'Avro is a serialization and RPC framework.', - license = 'Apache License 2.0', - keywords = 'avro serialization rpc', - url = 'http://hadoop.apache.org/avro', + name = 'avro-python3-snapshot', + version = avro_version, + packages = ['avro'], + package_dir = {'avro': 'avro'}, + scripts = ['scripts/avro'], + + package_data = { + 'avro': [ + 'HandshakeRequest.avsc', + 'HandshakeResponse.avsc', + VERSION_FILE_NAME, + ], + }, + + test_suite='avro.tests.run_tests', + tests_require=[], + + # metadata for upload to PyPI + author = 'Apache Avro', + author_email = 'avro-dev@hadoop.apache.org', + description = 'Avro is a serialization and RPC framework.', + license = 'Apache License 2.0', + keywords = 'avro serialization rpc', + url = 'http://hadoop.apache.org/avro', ) diff --git a/lang/ruby/Manifest b/lang/ruby/Manifest index 13daed9a..fa2c7283 100644 --- a/lang/ruby/Manifest +++ b/lang/ruby/Manifest @@ -4,7 +4,6 @@ Rakefile avro.gemspec interop/test_interop.rb lib/avro.rb -lib/avro/collect_hash.rb lib/avro/data_file.rb lib/avro/io.rb lib/avro/ipc.rb diff --git a/lang/ruby/lib/avro.rb b/lang/ruby/lib/avro.rb index de70e914..902dcd88 100644 --- a/lang/ruby/lib/avro.rb +++ b/lang/ruby/lib/avro.rb @@ -34,7 +34,6 @@ def initialize(schm=nil, datum=nil, msg=nil) end end -require 'avro/collect_hash' require 'avro/schema' require 'avro/io' require 'avro/data_file' diff --git a/lang/ruby/lib/avro/collect_hash.rb b/lang/ruby/lib/avro/collect_hash.rb deleted file mode 100644 index 49750807..00000000 --- a/lang/ruby/lib/avro/collect_hash.rb +++ /dev/null @@ -1,25 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -module Enumerable - def collect_hash - inject(Hash.new) do |memo, i| - k, v = yield(i) - memo[k] = v if k - memo - end - end -end diff --git a/lang/ruby/lib/avro/data_file.rb b/lang/ruby/lib/avro/data_file.rb index c2a6a526..6c794de4 100644 --- a/lang/ruby/lib/avro/data_file.rb +++ b/lang/ruby/lib/avro/data_file.rb @@ -20,7 +20,8 @@ module Avro module DataFile VERSION = 1 MAGIC = "Obj" + [VERSION].pack('c') - MAGIC_SIZE = MAGIC.size + MAGIC.force_encoding('BINARY') if MAGIC.respond_to?(:force_encoding) + MAGIC_SIZE = MAGIC.respond_to?(:bytesize) ? MAGIC.bytesize : MAGIC.size SYNC_SIZE = 16 SYNC_INTERVAL = 4000 * SYNC_SIZE META_SCHEMA = Schema.parse('{"type": "map", "values": "bytes"}') @@ -98,6 +99,7 @@ def initialize(writer, datum_writer, writers_schema=nil, codec=nil) @encoder = IO::BinaryEncoder.new(@writer) @datum_writer = datum_writer @buffer_writer = StringIO.new('', 'w') + @buffer_writer.set_encoding('BINARY') if @buffer_writer.respond_to?(:set_encoding) @buffer_encoder = IO::BinaryEncoder.new(@buffer_writer) @block_count = 0 @@ -181,7 +183,7 @@ def write_block # write number of items in block and block size in bytes encoder.write_long(block_count) to_write = codec.compress(buffer_writer.string) - encoder.write_long(to_write.size) + encoder.write_long(to_write.respond_to?(:bytesize) ? to_write.bytesize : to_write.size) # write block contents writer.write(to_write) diff --git a/lang/ruby/lib/avro/protocol.rb b/lang/ruby/lib/avro/protocol.rb index 321a4342..6c210e19 100644 --- a/lang/ruby/lib/avro/protocol.rb +++ b/lang/ruby/lib/avro/protocol.rb @@ -104,7 +104,7 @@ def to_avro(names=Set.new) hsh['types'] = types.map{|t| t.to_avro(names) } if types if messages - hsh['messages'] = messages.collect_hash{|k,t| [k, t.to_avro(names)] } + hsh['messages'] = messages.inject({}) {|h, (k,t)| h[k] = t.to_avro(names); h } end hsh diff --git a/pom.xml b/pom.xml index 8eeb880e..e188eb06 100644 --- a/pom.xml +++ b/pom.xml @@ -27,7 +27,7 @@ org.apache.avro avro-toplevel - 1.7.7-SNAPSHOT + 1.7.7 pom Apache Avro Toplevel diff --git a/share/VERSION.txt b/share/VERSION.txt index 0752e489..73c8b4f9 100644 --- a/share/VERSION.txt +++ b/share/VERSION.txt @@ -1 +1 @@ -1.7.7-SNAPSHOT \ No newline at end of file +1.7.7 \ No newline at end of file diff --git a/share/rat-excludes.txt b/share/rat-excludes.txt index c9e54978..c123a935 100644 --- a/share/rat-excludes.txt +++ b/share/rat-excludes.txt @@ -49,4 +49,7 @@ lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/*.css lang/java/protobuf/src/test/java/org/apache/avro/protobuf/Test.java lang/java/thrift/src/test/java/org/apache/avro/thrift/test/*.java lang/java/tools/src/test/compiler/output*/** +lang/perl/.shipit +lang/perl/Changes +lang/perl/MANIFEST* share/test/data/test.avro12