84 Star 748 Fork 277

GVPApache SeaTunnel / SeaTunnel

2023-12-04 11:51
1720668 nlfxwz 1641376644 Apache SeaTunnel

部署二进制请去官网:https://seatunnel.apache.org/download 下载!!!

2.3.3 Release Note

Bug fix

Core

  • [Translation] [Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type (#5170)

Connector

  • [Connector] [Jdbc] Fix the problem of JdbcOutputFormat database connection leak (#4802)
  • [Connector-V2] [AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146)
  • [Connector-V2] [CDC-base] Fix cdc connection does not close (#4922)
  • [Connector-V2] [Clickhouse] Fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (#5080)
  • [Connector-V2] [ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (#4937)
  • [Connector-V2] [Doris] Add streamload Content-type for doris URLdecode error (#4880)
  • [Connector-V2] [E2E] Fix maven scope (#4901)
  • [Connector-V2] [File-Base] Fix Hadoop Kerberos authentication related issues (#5171)
  • [Connector-V2] [Flink-runtime] Flink register table Environment The running mode is set tojob.mode (#4826)
  • [Connector-V2] [HbaseSink] Fix default timestamp (#4958)
  • [Connector-V2] [Hive] Support user-defined hive-site.xml (#4965)
  • [Connector-V2] [Jdbc] Support postgresql json type (#5194)
  • [Connector-V2] [Jdbc] Fix XA Transaction bug (#5020)
  • [Connector-V2] [Jdbc] Modify error comment in JdbcSourceConfigFactory (#5088)
  • [Connector-V2] [Jdbc-Oracle] Fix the Oracle number type mapping problem (#5209)
  • [Connector-V2] [Jdbc-Postgres] PostgreSQL versions below 9.5 are compatible use cdc sync problem (#5120)
  • [Connector-V2] [JindoOssFile] Fix plugin-mapping.properties (#5215)
  • [Connector-V2] [LocalFileSink] Fix LocalFile Sink file_format_type (#5118)
  • [Connector-V2] [Mongodb] Fix document error content and remove redundant code (#4982)
  • [Connector-V2] [Mongodb] Compatible with historical parameters (#4997)
  • [Connector-V2] [MongoDB-CDC] Incremental data kind error in snapshot phase (#5184)
  • [Connector-V2] [MongoDB-CDC] Solve startup resume token is negative (#5143)
  • [Connector-V2] [MongoDB-CDC] Refine data format to adapt to universal logic (#5162)
  • [Connector-V2] [MongoDB-CDC] Fix array index anomalies caused by #5057 (#5195)
  • [Connector-V2] [MySql-CDC] Fix jdbc connection leak for mysql (#5037)
  • [Connector-V2] [MySql-CDC] Fix mysql bit column to java byte (#4817)
  • [Connector-V2] [MySql-CDC] Fix listener not released when BinlogClient reuse (#5011)
  • [Connector-V2] [MySql-CDC] ServerId from int to long (#5033) (#5035)
  • [Connector-V2] [Paimon] Bump paimon-bundle version to 0.4.0-incubating (#5219)
  • [Connector-V2] [Redis] Fix redis nodes format error (#4981)
  • [Connector-V2] [StarRocks] Fix StarRocksJsonSerializer will transform array/map/row to string (#5281)
  • [Connector-V2] [StarRocks] Fix code style (#4966)

Zeta(ST-Engine)

  • [Zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0 (#5263)
  • [Zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (#5191)
  • [Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (#5152)
  • [Zeta] Fix MultipleTableJobConfigParser ignore env option (#5067)
  • [Zeta] Fix checkpoint error report without msg (#5137)
  • [Zeta] [REST-API] Fix class not found exception (#5063)
  • [Zeta] [REST-API] Fix browser long type intercept (#5267)
  • [Zeta] Fix task state memory leak (#5139)
  • [Zeta] Fix when execute seatunnel.sh, console may print error (#4984)
  • [Zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore (#5214)
  • [Zeta] Avoid Redundant Job Submissions by Checking Job Status (#5229)
  • [Zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation (#5206)
  • [Zeta] Fix after the savepoint job is restored, the checkpoint file cannot be generated #4985 (#5051)
  • [Zeta] Fixed multi-table job data loss and latency issues (#149) (#5031)
  • [Zeta] Fix the HdfsStorage can not delete checkpoint file #5046 (#5054)
  • [Zeta] Fix if the initialization task throws an error and the task can not return to master (#4888)

TEST

  • [Test] Split updated modules integration test for part 5 (#5208)
  • [Test] Add code style check when docs changed (#5183)
  • [Test] Turn on the testSavepoint test (#5199)
  • [Test] Fix the CI Job name error (#5032)
  • [Test] [E2E] Fix testGetErrorInfo case error (#5282)
  • [Test] [Kafka] Kafka e2e error (#5200)
  • [Test] [MongoDB-CDC] Fix mongodb cdc e2e instability (#5128)
  • [Test] Declare files that will always have UNIX line endings on checkout (#5221)
  • [Test] Fixed a minor bug (#5274)
  • [Test] Fixed IMap file storage e2e bug (#5237)
  • [Test] Split updated modules integration test for part 4 (#5028)
  • [Test] Fix zeta ci error (#5254)

Others

  • Update the project description (#4967)
  • Update version to 2.3.3-SNAPSHOT (#5043)
  • Fix com.google.common.base.Preconditions to seatunnel shade one (#5284)
  • Fix style error (#5280)
  • Fix code style (#5092)

Improve

Core

  • [API & Zeta] Using connector custom serializer encode/decode states (#5238)
  • [API]Improve savemode api (#4767)

Connector

  • [CDC Base] Add some split parameters to the optionRule (#5161)
  • [CDC Base] Support exactly-once of cdc and fix the BinlogOffset comparing bug (#5057)
  • [CDC Base] Add a fast sampling method that supports character types (#5179)
  • [CDC Base] Change driver scope to provider (#5002)
  • [CDC Base] Remove driver for cdc connector (#4952)
  • [Connector-V2] [File] Optimize files commit order (#5045)
  • [Connector-V2] [HiveSink] Remove drop partition when abort (#4940)
  • [Connector-V2] [Jdbc] Remove scheduler in JDBC sink #4736 (#5168)
  • [Connector-V2] [Jdbc] Check url not null throw friendly message (#5097)
  • [Connector-V2] [Jdbc] Support for Decimal types as splict keys (#4634)
  • [Connector-V2] [Mongodb] Sink support transaction update/writing (#5034)
  • [Connector-V2] [Mongodb] Optimize reading logic (#5001)
  • [Connector-V2] [Neo4j] Supports neo4j sink batch write and update docs (#4841)
  • [Connector-V2] [OSS-Jindo] Optimize jindo oss connector (#4964)
  • [Connector-V2] [Tablestore sink] Remove scheduler in Tablestore sink (#5272)

Transform

  • [Transform-V2] [SQL] Support use catalogTableName as SQL expression (#5273)

CI

  • [CI/CD] Remove 'paths-ignore', enable the code style check for markdown files (#5286)

Zeta(ST-Engine)

  • [Zeta] Upgrade the key log output level (#4993)
  • [Zeta] Don't trigger handleSaveMode when restore (#5192)
  • [Zeta] Move checkpoint notify complete in checkpoint stage (#5185)
  • [Zeta] Add sleep for Task to reduce CPU cost (#5117)
  • [Zeta] Add an interface for batch retrieval of JobMetrics (#4576)
  • [Zeta] Move imap storage file dependency packages to submodules (#5218)
  • [Zeta] Improve CheckpointCoordinator log error when report error from task (#178) (#5134)
  • [Zeta] Improve CheckpointCoordinator notify complete when restore (#5136)

Feature

Connector

  • [Connector-V2] [CDC-base] Support tables without primary keys (with unique keys) (#163) (#5150)
  • [Connector-V2] [CDC-base] Support disable/enable exactly once for INITIAL (#4921)
  • [Connector-V2] [CDC-base] Change the time zone to the default time zone (#5030)
  • [Connector-V2] [CDC-base] Support string type shard fields (#5147)
  • [Connector-V2] [Clickhouse] Add clickhouse connector time zone key,default system time zone (#5078)
  • [Connector-V2] [Clickhouse] Clickhouse writes with checkpoints (#4999)
  • [Connector-V2] [Doris] Add doris factory type convertor (#5061)
  • [Connector-V2] [File] Add config of 'file_filter_pattern', which used for filtering files (#5153)
  • [Connector-V2] [File] Add cos source&sink (#4979)
  • [Connector-V2] [Hive] Hive add option because web need (#5154)
  • [Connector-V2] [Jdbc] Add JDBC Catalog auto create table (#4917)
  • [Connector-V2] [Jdbc] Add oceanbase dialect factory (#4989)
  • [Connector-V2] [Jdbc] JDBC source support string type as partition key (#4947)
  • [Connector-V2] [Jdbc-DM] Add DM upsert support (#5073)
  • [Connector-V2] [Kafka] Support read debezium format message from kafka (#5066)
  • [Connector-V2] [Kafka] Connector-kafka source support data conversion extracted by kafka connect source (#4516)
  • [Connector-V2] [Mongodb] Mongodb support cdc sink (#4833)
  • [Connector-V2] [Mongodb-CDC] Support source mongodb cdc (#4923)
  • [Connector-V2] [Mysql-CDC] Conversion of tinyint(1) to bool is supported (#5105)
  • [Connector-V2] [Redis] Support redis to set expiration time (#4975)

Zeta(ST-Engine)

  • [Zeta] [CDC-base] Support schema evolution framework(DDL) (#5125)
  • [Zeta] [REST-API] Add REST API To Submit Job (#5107)
  • [Zeta] Checkpoint support hdfs ha mode (#4942)
  • [Zeta] The expiration time of a historical Job can be config (#5180)
  • [Zeta] Support history service record job execute error (#5114)
  • [Zeta] Configuration files support user variable replacement (#4969)

Transform

  • [Transform-V2] [SQL] Support 'select *' and 'like' clause for SQL Transform plugin (#4991)

CI

  • [CI] Update action config to support run CI on fork repo (#5065)
  • [E2E] [FtpFile] Add ftp file e2e test case (#4647)

Docs

  • [Connector-V2][Clickhouse] Reconstruct the clickhouse connector doc (#5085)
  • [Connector-V2][Doris] Reconstruct the Doris connector document (#4903)
  • [Connector-V2][HDFS] Refactor connector-v2 docs using unified format HDFS (#4871)
  • [Connector-V2][Http] Reconstruct the Http connector document (#4962)
  • [Connector-V2][Hudi] Reconstruct the Hudi connector document (#4905)
  • [Connector-V2][Jdbc] Change the line boundary store value type to BigDecimal (#4900)
  • [Connector-V2][Jdbc-DB2、Jdbc-Vertica] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (#5102)
  • [Connector-V2][Jdbc-Oracle] Add JDBC Oracle Documentation (#5239)
  • [Connector-V2][Kafka] Reconstruct the kafka connector document (#4778)
  • [Connector-V2][MyHours] Reconstruct the MyHours connector document (#5129)
  • [Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL #4590 (#4757)
  • [Connector-V2][S3File] Improve S3File Source & S3File Sink document (#5101)
  • [Connector-V2][SelectDB-Cloud] Reconstruct the SelectDB-Cloud connector document (#5130)
  • [Connector-V2][StarRocks] Reconstruct the StarRocks connector document (#5132)
  • [Connector-V2][StarRocks] StarRocks nodeUrls property name fix (#4951)
  • [README] Remove useless github workflow, and adjust description of 'engineering structure' (#4305)
  • [Zeta] Add savepoint doc (#5081)
  • Update bug-report.yml (#5160)
  • Add Value types in Java to Schema feature (#5087)
  • Documentation and partial word optimization (#4936)
  • Change repository name from incubator-seatunnel to seatunnel (#4868)
  • Update .asf.yaml (#5242)
  • Update the Readme (#4968)
  • Set Up with Kubernetes, dockerfile document error in constructing docker image (#5022)
  • Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (#4793)
  • Change the transform website url (#4954)
  • Update quick-start-seatunnel-engine.md (#4943)
  • Remove incubating keyword in document (#5257)
2023-06-22 22:55
1720668 nlfxwz 1641376644 Apache SeaTunnel

2.3.2 Release Note

Bug fix

Core

  • [Core] [API] Fixed generic class loss for lists (#4421)
  • [Core] [API] Fix parse nested row data type key changed upper (#4459)

Connector-V2

  • [Json-format] [Canal-Json] Fix json deserialize NPE (#4195)
  • [Connector-V2] [Jdbc] Field aliases are not supported in the query of jdbc source. (#4210)
  • [Connector-V2] [Jdbc] Fix connection failure caused by connection timeout. (#4322)
  • [Connector-V2] [Jdbc] Set default value to false of JdbcOption: generate_sink_sql (#4471)
  • [Connector-V2] [JDBC] Fix TiDBCatalog without open (#4718)
  • [Connector-V2] [Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (#4866)
  • [Connector-V2] [Pulsar] Fix the bug that can't consume messages all the time. (#4125)
  • [Connector-V2] [Eleasticsearch] Document description error (#4390)
  • [Connector-V2] [Eleasticsearch] Source deserializer error and inappropriate (#4233)
  • [Connector-V2] [Kafka] Fix KafkaProducer resources have never been released. (#4302)
  • [Connector-V2] [Kafka] Fix the permission problem caused by client.id. (#4246)
  • [Connector-V2] [Kafka] Fix KafkaConsumerThread exit caused by commit offset error. (#4379)
  • [Connector-V2] [kafka] Fix the problem that the partition information can not be obtained when kafka is restored (#4764)
  • [Connector-V2] [SFTP] Fix incorrect exception handling logic (#4720)
  • [Connector-V2] [File] Fix read temp file (#4876)
  • [Connector-V2] [CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (#4877)
  • [Connector-V2] [Doris] update last checkpoint id when doing snapshot (#4881)
  • [Connector-v2] [kafka] Fix the short interval of pull data settings and revise the format (#4875)
  • [Connector-v2] [RabbitMQ] Fix reduplicate ack msg bug and code style (#4842)
  • [Connector-V2] [Jdbc] Fix the error of extracting primary key column in sink (#4815)
  • [Connector-V2] [Jdbc] Fix reconnect throw close statement exception (#4801)
  • [Connector-V2] [Jdbc] Fix sqlserver system table case sensitivity (#4806)
  • [Connector-v2] [File] Fix configuration file format and error comments (#4762)
  • [Connector-v2] [Jdbc] Fix oracle sql table identifier (#4754)
  • [Connector-v2] [Clickhouse] fix get clickhouse local table name with closing bracket from distributed table engineFull (#4710)

Zeta(ST-Engine)

  • [Zeta] Fix LogicalDagGeneratorTest testcase (#4401)
  • [Zeta] Fix MultipleTableJobConfigParser parse only one transform (#4412)
  • [Zeta] Fix missing common plugin jars (#4448)
  • [Zeta] Fix handleCheckpointError be called while checkpoint already complete (#4442)
  • [Zeta] Fix job error message is not right bug (#4463)
  • [Zeta] Fix finding TaskGroup deployment node bug (#4449)
  • [Zeta] Fix the bug of conf (#4488)
  • [Zeta] Fix Connector load logic from zeta (#4510)
  • [Zeta] Fix conflict dependency of hadoop-hdfs (#4509)
  • [Zeta] Fix TaskExecutionService synchronized lock will not release (#4886)
  • [Zeta] Fix TaskExecutionService will return not active ExecutionContext (#4869)
  • [Zeta] Fix deploy operation timeout but task already finished bug (#4867)
  • [Zeta] Fix restoreComplete Future can't be completed when cancel task (#4863)
  • [Zeta] Fix IMap operation timeout bug (#4859)
  • [Zeta] fix pipeline state not right bug (#4823)
  • [Zeta] Fix the incorrect setting of transform parallelism (#4814)
  • [Zeta] Fix master active bug (#4855)
  • [Zeta] Fix completePendingCheckpoint concurrent action (#4854)
  • [Zeta] Fix engine runtime error (#4850)
  • [Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (#4849)
  • [Zeta] Fix task notifyTaskStatusToMaster failed when job not running or failed before run (#4847)
  • [Zeta] Fix cpu load problem (#4828)
  • [zeta] Fix the deadlock issue with JDBC driver loading (#4878)

E2E

  • [E2E] [Kafka] Fix kafka e2e testcase (#4520)
  • [Container Version] Fix risk of unreproducible test cases #4591
  • [E2e] [Mysql-cdc] Removing the excess MySqlIncrementalSourceIT e2e reduces the CI time (#4738)

Improve

Core

  • [Core] [Spark] Push transform operation from Spark Driver to Executors (#4503)
  • [Core] [Starter] Optimize code structure & remove redundant code (#4525)
  • [Core] [Translation] [Flink] Optimize code structure & remove redundant code (#4527)
  • [Core] [Starter] Add check of sink and source config to avoid null pointer exception. (#4734)

Connector-V2

  • [Connector-V2] [CDC] Improve startup.mode/stop.mode options (#4360)
  • [Connector-V2] [CDC] Optimize jdbc fetch-size options (#4352)
  • [Connector-V2] [CDC] Fix chunk start/end parameter type error (#4777)
  • [Connector-V2] [SQLServer] Fix sqlserver catalog (#4441)
  • [Connector-V2] [StarRocks] Improve StarRocks Serialize Error Message (#4458)
  • [Connector-V2] [Jdbc] add the log for sql and update some style (#4475)
  • [Connector-V2] [Jdbc] Fix the table name is not automatically obtained when multiple tables (#4514)
  • [Connector-V2] [S3 & Kafka] Delete unavailable S3 & Kafka Catalogs (#4477)
  • [Connector-V2] [Pulsar] Support Canal Format
  • [Connector-V2] [CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (#4856)
  • [Connector-V2] [SelectDB] Add a jobId to the selectDB label to distinguish between tasks (#4864)
  • [Connector-V2] [Doris] Add a jobId to the doris label to distinguish between tasks (#4839) (#4853)
  • [Connector-v2] [Mongodb]Refactor mongodb connector (#4620)
  • [Connector-v2] [Jdbc] Populate primary key when jdbc sink is created using CatalogTable (#4755)
  • [Transform-V2] Optimize SQL Transform package and Fix Spark type conversion bug of transform (#4490)

CI

  • [CI] Fix error repository name in ci config files (#4795)

Zeta(ST-Engine)

  • [Zeta] Support run the server through daemon mode (#4161)
  • [Zeta] Change ClassLoader To Improve the SDK compatibility of the client (#4447)
  • [Zeta] Client Support Async Submit Job (#4456)
  • [Zeta] Add more detailed log output. (#4446)
  • [Zeta] Improve seatunnel-cluster.sh (#4435)
  • [Zeta] Reduce CPU Cost When Task Not Ready (#4479)
  • [Zeta] Add parser log (#4485)
  • [Zeta] Remove redundant code (#4489)
  • [Zeta] Remove redundancy code in validateSQL (#4506)
  • [Zeta] Improve JobMetrics fetch performance (#4467)
  • [Zeta] Reduce the operation count of imap_running_job_metrics (#4861)
  • [Zeta] Speed up listAllJob function (#4852)
  • [Zeta] async execute checkpoint trigger and other block method (#4846)
  • [Zeta] Reduce the number of IMAPs used by checkpointIdCounter (#4832)
  • [Zeta] Cancel pipeline add retry to avoid cancel failed. (#4792)
  • [Zeta] Improve Zeta operation max count and ignore NPE (#4787)
  • [Zeta] Remove serialize(deserialize) cost when use shuffle action (#4722)

Feature

Core

  • [Core] [API] Support convert strings as List option (#4362)
  • [Core] [API] Add copy method to Catalog codes (#4414)
  • [Core] [API] Add options check before create source and sink and transform in FactoryUtil (#4424)
  • [Core] [Shade] Add guava shade module (#4358)

Connector-V2

  • [Connector-V2] [CDC] [SQLServer] Support multi-table read (#4377)
  • [Connector-V2] [Kafka] Kafka source supports data deserialization failure skipping (#4364)
  • [Connector-V2] [Jdbc] [TiDB] Add TiDB catalog (#4438)
  • [Connector-V2] [File] Add file excel sink and source (#4164)
  • [Connector-v2] [Snowflake] Add Snowflake Source&Sink connector (#4470)
  • [Connector-V2] [Pular] support read format for pulsar (#4111)
  • [Connector-V2] [Paimon] Introduce paimon connector (#4178)
  • [Connector V2] [Cassandra] Expose configurable options in Cassandra (#3681)
  • [Connector V2] [Jdbc] Supports GEOMETRY data type for PostgreSQL (#4673)
  • [Transform-V2] Add UDF SPI and an example implement for SQL Transform plugin (#4392)
  • [Transform-V2] Support copy field list (#4404)
  • [Transform-V2] Add support CatalogTable for FieldMapperTransform (#4423)
  • [Transform-V2] Add CatalogTable support for ReplaceTransform (#4411)
  • [Transform-V2] Add Catalog support for FilterRowKindTransform (#4420)
  • [Transform-V2] Add support CatalogTable for FilterFieldTransform (#4422)
  • [Transform-V2] Add catalog support for SQL Transform plugin (#4819)

Zeta(ST-Engine)

  • [Zeta] Support for mixing Factory and Plugin SPI (#4359)
  • [Zeta] Add get running job info by jobId rest api (#4140)
  • [Zeta] Add REST API To Get System Monitoring Information (#4315)
  • [Transform V2 & Zeta] Make SplitTransform Support CatalogTable And CatalogTable Evolution (#4396)
  • [Zeta] Move driver into lib directory and change operation count (#4845)
  • [Zeta] Add Metaspace size default value to config file (#4848)
  • [Zeta] Reduce the frequency of fetching data from imap (#4851)
  • [Zeta] Add OSS support for Imap storage to cluster-mode type (#4683)
  • [Zeta] Improve local mode startup request ports (#4660)

Docs

  • [Docs] Optimizes part of the Doris and SelectDB connector documentation (#4365)
  • [Docs] Fix docs code style (#4368)
  • [Docs] Update jdbc doc and kafka doc (#4380)
  • [Docs] Fix max_retries default value is 0. (#4383)
  • [Docs] Fix markdown syntax (#4426)
  • [Docs] Fix Kafka Doc Error Config Key "kafka." (#4427)
  • [Docs] Add Transform to Quick Start v2 (#4436)
  • [Docs] Fix Mysql sink format doc (#4800)
  • [Docs] Add the generate sink sql parameter for the jdbc sink document (#4797)
  • [Docs] Add the generate sink sql parameter And example (#4769)
  • [Docs] Redshift add defaultRowFetchSize (#4616)
  • [Docs] Refactor connector-v2 docs using unified format Mysql (#4590)
2023-01-16 13:45
1720668 nlfxwz 1641376644 Apache SeaTunnel

2.3.0-release

Bug fix

Core

  • [Core] [Starter] Fix the bug of ST log print failed in some jdk versions #3160
  • [Core] [Shell] Fix bug that shell script about downloading plugins does not work #3462

Connector-V2

  • [Connector-V2] [Jdbc] Fix the bug that jdbc source can not be stopped in batch mode #3220
  • [Connector-V2] [Jdbc] Fix the bug that jdbc connector reset in jdbc connector #3670
  • [Connector-V2] [Jdbc] Fix the bug that jdbc connector exactly-once it will throw NullPointerException #3730
  • [Connector-V2] [Hive] Fix the following bugs of hive connector: 1. write parquet NullPointerException 2. when restore write from states getting error file path #3258
  • [Connector-V2] [File] Fix the bug that when getting file system throw NullPointerException #3506
  • [Connector-V2] [File] Fix the bug that when user does not config the fileNameExpression it will throw NullPointerException #3706
  • [Connector-V2] [Hudi] Fix the bug that the split owner of Hudi connector may be negative #3184

ST-Engine

  • [ST-Engine] Fix bug data file name will duplicate when use SeaTunnel Engine #3717
  • [ST-Engine] Fix job restart of all nodes down #3722
  • [ST-Engine] Fix the bug that checkpoint stuck in ST-Engine #3213
  • [ST-Engine] Fix the bug that checkpoint failed in ST-Engine #3769

E2E

  • [E2E] [Spark] Corrected spark version in e2e container #3225

Improve

Core

  • [Core] [Starter] [Flink] Upgrade the method of loading extra jars in flink starter #2982
  • [Core] [Pom] [Package] Optimize package process #3751

Connector-V1

  • [Connector-V1] Remove connector v1 related codes from dev branch #3450

Connector-V2

  • [Connector-V2] Add split templates for all connectors #3335
  • [Connector-V2] [Redis] Support redis cluster mode & user authentication #3188
  • [Connector-V2] [Clickhouse] Support nest type and array type in clickhouse connector #3047
  • [Connector-V2] [Clickhouse] Support geo type in clickhouse connector #3141
  • [Connector-V2] [Clickhouse] Improve double convert that in clickhouse connector #3441
  • [Connector-V2] [Clickhouse] Improve float long convert that in clickhouse connector #3471
  • [Connector-V2] [Kafka] Support setting read start offset or message time in kafka connector #3157
  • [Connector-V2] [Kafka] Support specify multiple partition keys in kafka connector #3230
  • [Connector-V2] [Kafka] Support dynamic discover topic & partition in kafka connector #3125
  • [Connector-V2] [Kafka] Support text format for kafka connector #3711
  • [Connector-V2] [IotDB] Add the parameter check logic for iotDB sink connector #3412
  • [Connector-V2] [Jdbc] Support setting fetch size in jdbc connector #3478
  • [Connector-V2] [Jdbc] Support upsert config in jdbc connector #3708
  • [Connector-V2] [Jdbc] Optimize the commit process of jdbc connector #3451
  • [Connector-V2] [Jdbc] Release jdbc resource when after using #3358
  • [Connector-V2] [Oracle] Improve data type mapping of Oracle connector #3486
  • [Connector-V2] [Http] Support extract complex json string in http connector #3510
  • [Connector-V2] [File] [S3] Support s3a protocol in S3 file connector #3632
  • [Connector-V2] [File] [HDFS] Support setting hdfs-site.xml #3778
  • [Connector-V2] [File] Support file split in file connectors #3625
  • [Connector-V2] [CDC] Support write cdc changelog event in elsticsearch sink connector #3673
  • [Connector-V2] [CDC] Support write cdc changelog event in clickhouse sink connector #3653
  • [Conncetor-V2] [CDC] Support write cdc changelog event in jdbc connector #3444

ST-Engine

  • [ST-Engine] Improve statistic information print format that in ST-Engine #3492
  • [ST-Engine] Improve ST-Engine performance #3216
  • [ST-Engine] Support user-defined jvm parameters in ST-Engine #3307

CI

  • [CI] Improve CI process #3179 #3194

E2E

  • [E2E] [Flink] Support execute extra commands on task-manager container #3224
  • [E2E] [Jdbc] Increased Jdbc e2e stability #3234

Feature

Core

  • [Core] [Log] Integrate slf4j and log4j2 for unified management logs #3025
  • [Core] [Connector-V2] [Exception] Unified exception API & Unified connector error tip message #3045
  • [Core] [Shade] [Hadoop] Add hadoop shade package for SeaTunnel #3755

Connector-V2

  • [Connector-V2] [Elasticsearch] Add elasticsearch source connector #2821
  • [Connector-V2] [AmazondynamoDB] Add AmazondynamoDB source & sink connector #3166
  • [Connector-V2] [StarRocks] Add StarRocks sink connector #3164
  • [Connector-V2] [DB2] Add DB2 source & sink connector #2410
  • [Connector-V2] [Transform] Add transform-v2 api #3145
  • [Connector-V2] [InfluxDB] Add influxDB sink connector #3174
  • [Connector-V2] [Cassandra] Add Cassandra Source & Sink connector #3229
  • [Connector-V2] [MyHours] Add MyHours source connector #3228
  • [Connector-V2] [Lemlist] Add Lemlist source connector #3346
  • [Connector-V2] [CDC] [MySql] Add mysql cdc source connector #3455
  • [Connector-V2] [CDC] [SqlServer] Add sqlserver cdc source connector #3686
  • [Connector-V2] [Klaviyo] Add Klaviyo source connector #3443
  • [Connector-V2] [OneSingal] Add OneSingal source connector #3454
  • [Connector-V2] [Slack] Add slack sink connector #3226
  • [Connector-V2] [Jira] Add Jira source connector #3473
  • [Connector-V2] [Sqlite] Add Sqlite source & sink connector #3089
  • [Connector-V2] [OpenMldb] Add openmldb source connector #3313
  • [Connector-V2] [Teradata] Add teradata source & sink connector #3362
  • [Connector-V2] [Doris] Add doris source & sink connector #3586
  • [Connector-V2] [MaxCompute] Add MaxCompute source & sink connector #3640
  • [Connector-V2] [Doris] [Streamload] Add doris streamload sink connector #3631
  • [Connector-V2] [Redshift] Add redshift source & sink connector #3615
  • [Connector-V2] [Notion] Add notion source connector #3470
  • [Connector-V2] [File] [Oss-Jindo] Add oss jindo source & sink connector #3456

ST-Engine

  • [ST-Engine] Support print job metrics when job finished #3691
  • [ST-Engine] Add metrics statistic in ST-Engine #3621
  • [ST-Engine] Support IMap file storage in ST-Engine #3418
  • [ST-Engine] Support S3 file system for IMap file storage #3675
  • [ST-Engine] Support save job restart status information in ST-Engine #3637

E2E

  • [E2E] [Http] Add http type connector e2e test cases #3340
  • [E2E] [File] [Local] Add local file connector e2e test cases #3221

Docs

  • [Docs] [Connector-V2] [Factory] Add TableSourceFactory & TableSinkFactor docs #3343
  • [Docs] [Connector-V2] [Schema] Add connector-v2 schema docs #3296
  • [Docs] [Connector-V2] [Quick-Manaul] Add error quick reference manual #3437
  • [Docs] [README] Improve README and refactored other docs #3619
2022-08-23 14:40
1720668 nlfxwz 1641376644 Apache SeaTunnel

[Feature & Improvement]
[Connector][Flink][Fake] Supported BigInteger Type (#2118)
[Connector][Spark][TiDB] Refactored config parameters (#1983)
[Connector][Flink]add AssertSink connector (#2022)
[Connector][Spark][ClickHouse]Support Rsync to transfer clickhouse data file (#2074)
[Connector & e2e][Flink] add IT for Assert Sink in e2e module (#2036)
[Transform][Spark] data quality for null data rate (#1978)
[Transform][Spark] Add a module to set default value for null field #1958
[Chore]a more understandable code,and code warning will disappear #2005
[Spark] Use higher version of the libthrift dependency (#1994)
[Core][Starter] Change jar connector load logic (#2193)
[Core]Add plugin discovery module (#1881)

[BUG]
[Connector][Hudi] Source loads the data twice
[Connector][Doris]Fix the bug Unrecognized field "TwoPhaseCommit" after doris 0.15 (#2054)
[Connector][Jdbc]Fix the data output exception when accessing Hive using Spark JDBC #2085
[Connector][Jdbc]Fix JDBC data loss occurs when partition_column (partition mode) is set #2033
[Connector][Kafka]KafkaTableStream schema json parse #2168
[seatunnel-core] Failed to get APP_DIR path bug fixed (#2165)
[seatunnel-api-flink] Connectors dependencies repeat additions (#2207)
[seatunnel-core] Failed to get APP_DIR path bug fixed (#2165)
[seatunnel-core-flink] Updated FlinkRunMode enum to get the proper help message for run modes. (#2008)
[seatunnel-core-flink]fix same source and sink registerplugin librarycache error (#2015)
[Command]fix commandArgs -t(--check) conflict with flink deployment target (#2174)
[Core][Jackson]fix jackson type convert error (#2031)
[Core][Starter] When use cluster mode, but starter app root dir also should same as client mode. (#2141)

Docs
source socket connector docs update (#1995)
Add uuid, udf, replace transform to doc (#2016)
Update Flink engine version requirements (#2220)
Add Flink SQL module to website. (#2021)
[kubernetes] update seatunnel doc on kubernetes (#2035)

Dependency upgrade
Upgrade common-collecions4 to 4.4
Upgrade common-codec to 1.13

2022-03-23 10:52
1720668 nlfxwz 1641376644 Apache SeaTunnel
  • Use JCommander to do command line parameter parsing, making developers focus on the logic itself.
  • Flink is upgraded from 1.9 to 1.13.5, keeping compatibility with older versions and preparing for subsequent CDC.
  • Support for Doris, Hudi, Phoenix, Druid, and other Connector plugins, and you can find complete plugin support here plugins-supported-by-seatunnel.
  • Local development extremely fast starts environment support. It can be achieved by using the example module without modifying any code, which is convenient for local debugging.
  • Support for installing and trying out Apache SeaTunnel(Incubating) via Docker containers.
  • SQL component supports SET statements and configuration variables.
  • Config module refactoring to facilitate understanding for the contributors while ensuring code compliance (License) of the project.
  • Project structure realigned to fit the new Roadmap.
  • CI&CD support, code quality automation control (more plans will be carried out to support CI&CD development).
  • Please go to the official channel to download: https://seatunnel.apache.org/download
2022-01-05 22:46
1720668 nlfxwz 1641376644 Apache SeaTunnel

published api modules to maven central repo.

added waterdrop-config source code.

added build.md guide.

refined project code and pom.xml structure.

2022-01-05 22:44
1720668 nlfxwz 1641376644 Apache SeaTunnel
  • [project rename] changed start-waterdrop.sh to start-seatunnel.sh, changed logo ascii code from waterdrop to seatunnel by @garyelephant
  • [Feature] added the abstraction of BaseAction by @garyelephant in #810
  • [feature] allow user to customize log4j.properties @garyelephant in #267 (comment)
  • [bugfix] fixed a bug of kerberos config in spark config by @garyelephant in #590
  • [bugfix] Fix bug of #719 by @rickyhuo in #743
最后提交信息为: bump version to 1.5.6
2022-01-05 22:39
1720668 nlfxwz 1641376644 Apache SeaTunnel
最后提交信息为: bump version to v1.5.7
Java
1
https://gitee.com/seatunnel/SeaTunnel.git
git@gitee.com:seatunnel/SeaTunnel.git
seatunnel
SeaTunnel
SeaTunnel

搜索帮助

53164aa7 5694891 3bd8fe86 5694891