High-dimensional quantum units of information, or qudits, can carry more than one quantum bit of information in a single degree of freedom and can, therefore, be used to boost the performance of quantum communication and quantum computation protocols. A photon in a superposition of 2^N time bins—a time-bin qudit—contains as much information as N qubits. Here, we show that N-qubit states encoded in a single time-bin qudit can be arbitrarily and deterministically generated, manipulated, and measured using a number of linear optics elements that scale linearly with N, as opposed to prior proposals of single-qudit implementation of N-qubit logic, which typically requires O(2^N) elements. The simple and cost-effective implementation we propose can be used as a small-scale quantum processor. We then demonstrate a path toward scalability by interfacing distinct qudit processors to a matter qubit (atom or quantum dot spin) in an optical resonator. Such a cavity quantum electrodynamics system allows for more advanced functionalities, such as single-qubit nondemolition measurement and two-qubit gates between distinct qudits. It could also enable quantum interfaces with other matter quantum nodes in the context of quantum networks and distributed quantum computing.