The bulk RNA-seq technology allows researchers to find differentially expressed genes or alternative splicing variants. Multiple methods for investigating these problems have been developed so far, however their performance can only be reliably evaluated on synthetic data or by performing expensive spike-in experiments. Because of the need for such evaluations, various methods for generating synthetic bulk RNA-seq data have been developed. Those methods deploy parametric, semiparametric, or nonparametric approaches to generate datasets based on different input data or parameters. Currently, there is no complete and systematic approach for evaluating different characteristics and performance metrics of such data-generation methods, especially in terms of “closeness” to the original data. In this work, we present an initial framework for comparing different aspects of the data generated by several of the currently available algorithms for synthetic bulk RNA-seq data generation. We demonstrate that there are noticeable differences between those data generators, even in cases when they use the same input data set. We also propose several metrics that could be used as components of a systematic approach for comparative evaluation of algorithms for synthetic bulk RNA-seq data generation.